Freelance Senior Software Engineer | AI Evaluation

Навыки

• AI-агенты
• AI-инструменты в работе
• DevTools
• Docker
• FastAPI
• JavaScript
• Kafka

О компании и продукте

• We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks. Essentially, we're building an olympiad for AI models: the better the tasks, the more precisely we see where models are weak, and the faster they improve.
• You create tasks and the criteria to evaluate them. Each task consists of:
• An isolated environment - an emulation of a developer's workstation: a Linux machine with dev tools (terminal, CLI), MCP servers (repository, task tracker, messenger, documentation, etc.), and a real web application codebase
• A task - a prompt, just like in real work (a ticket link, a messenger conversation, anything). The agent figures it out on its own - reads the ticket, checks discussions, searches docs, explores the code. All of that context is also part of the task and needs to be created and thought through

Задачи

• Company Intro
• At Toloka AI we create data that powers leading GenAI models and innovations
• We work with frontier labs, big tech, renowned AI startups, enterprises and non-profit research organizations worldwide
• We use a combination of Experts + Crowd + Tech Platform to teach AI models to reason and evaluate their efficacy and safety
• We have experts in more than 50 different domains—from doctors and lawyers to physicists and engineers and boast one of the most diverse global crowds, representing over 100 countries and speaking 40+ languages

Требования

• Strong software engineering experience: 8+ years in any modern programming language, with a willingness and ability to work primarily in Python in our projects
• Hands-on experience using coding agents such as Claude Code and Codex as part of practical engineering workflows, while maintaining ownership of architecture, correctness, code quality, and delivery
• Problem-solving and clarity of thinking
• Ability to communicate technical decisions and reasoning
• Practical understanding of Docker and development workflows
• Overall readiness to contribute in a technical, execution-focused role
• What we can offer
• Freelance collaboration via the Ontop freelance platform
• Flexible, fully remote schedule ( 40 hours per week )
• You’ll have the opportunity to work on highly innovative projects at the leading edge of AI development, together with a genuinely dedicated and dynamic team of experts
• You’ll also be working on projects with customers that are AI industry leaders and well-known household names
• Friendly community
• Important Notice] Scam Alert Regarding Fake Job Postings
• It has come to our attention that an individual or group is fraudulently impersonating Toloka to post fake jobs and solicit personal information from applicants.Please be aware
• Official Communication: Our recruiting team will only contact you from an official " toloka.ai " email address
• We will NEVER use Gmail, Yahoo, Tolokainc, toloka.inc, or other personal or seemingly business email accounts
• Our Process: We will never ask for your bank account details, credit card number, or any fees as part of the application or interview process
• Official Listings: All legitimate job openings are posted on our official careers page: https://toloka.ai/careers#job-list
• What to do: If you see a suspicious job posting or have been contacted by someone you suspect is a scammer, please do not provide any personal information
• Instead, report the incident to us directly at security@toloka.ai and report the profile/post to LinkedIn.We are taking this matter very seriously and are working with the appropriate parties to resolve it
• Thank you for your vigilance!

Условия

• Our remote-first team is globally distributed around the world: USA , UK , the Netherlands , Serbia , and more

Паспорт вакансии

История публикации

Откуда что взялось

Отмечено то, что вывели мы. Без пометки — значение назвал работодатель.

Почему на этом месте в выдаче

Порядок выдачи объявлен контрактом: свежесть решает между днями, полнота и зарплата — внутри дня.