AIWorkJobs
Mercor logoMercorPlatform listing

Applied Engineer, Evaluations

Build and refine software engineering evaluations from finished pull requests, then run them against frontier coding agents such as Claude Code and Codex. You’ll use an in-house framework alongside a research team that…

Remote · Eligibility not specifiedPart-time
Work arrangement
Remote
Eligibility
Eligibility not specified
Engagement
Part-time
Compensation
$100–$120/hr
Specialty
Software Engineering

About the role

Build and refine software engineering evaluations from finished pull requests, then run them against frontier coding agents such as Claude Code and Codex. You’ll use an in-house framework alongside a research team that creates benchmarks for comparing large language models. The work covers repository selection, task design, automated grading, and repeated validation. Onboarding and ongoing review feedback support your start; afterward, you’ll own the work without step-by-step direction.

What you would do

  • Choose substantial repositories and completed pull requests that can support challenging engineering tasks.
  • Study each problem, its reference implementation, and the repository’s architecture, tests, and conventions.
  • Write task prompts and build evaluations with automated tests, shell commands, and LLM grading prompts.
  • Check evaluations against reference solutions and intentionally flawed implementations. Find missing checks, inaccurate grades, and criteria that restrict valid approaches without good reason.
  • Run evaluations across several models and harnesses, then determine whether failures stem from the agent’s code, the environment, or the grading.
  • Confirm that each task reveals meaningful weaknesses in agent performance.
  • Improve evaluations through repeated testing and review, document findings and grading decisions, and incorporate feedback.

Requirements

  • Strong programming fundamentals and hands-on experience with substantial, complex codebases.
  • Ability to learn unfamiliar code, investigate subtle behavior, and judge whether different implementations solve the same problem correctly.
  • Experience writing meaningful tests, including edge-case and regression coverage.
  • Practical experience with AI coding tools and the judgment to check their output for errors.
  • Confidence with Git, test runners, and CI tools.
  • Clear written and verbal communication, with the ability to manage a task independently and raise questions when requirements lack clarity.
  • Ability to work with evaluation tasks in languages such as TypeScript, JavaScript, Python, Java, Kotlin, Go, Ruby, PHP, C++, or Rust.

Who should apply

Software engineers with experience in complex repositories and a strong grasp of testing can bring sound judgment to evaluation design. Experience may come from open-source, private, or enterprise codebases; familiarity with a specific language or ecosystem helps, but public contributions and repository popularity matter less than the ability to learn the code.

How applying works on Mercor

Create a profile, complete the role checklist and AI interview, then verify your identity before a project offer.

  1. Create an account and confirm your résumé.
  2. Complete the role checklist, written assessment, and AI interview where the listing requires them.
  3. Submit the application and wait for review or matching.
  4. Complete a work trial and identity verification before onboarding.

Typical timeline: The AI interview is about 20 minutes; work trials are often 4–6 hours; identity checks are usually minutes.

Paid steps: The work trial is paid.

Assessment and interview requirements vary by role. Check the checklist inside the specific listing.

Process checked Nov 19, 2026. AIWorkJobs does not submit applications or charge a fee. Read the full Mercor process and its official source on the Mercor page.

Skills mentioned by the source

  • typescript
  • javascript
  • python
  • java
  • kotlin
  • go
  • ruby
  • php
  • c++
  • rust
  • git
  • automated testing
  • regression testing
  • ci
  • shell scripting
  • llm evaluation

Specialty areas

Specialties come from the source platform’s category taxonomy and are used to filter the job index.

Listing reference: applied-engineer-evaluations-NNEa6l. Source: Mercor. If something here contradicts the source, the source controls. Report a correction.

Related roles

Other work on the index.

See more in this category

Turing

Senior Software Engineer – LLM Evaluation

Evaluate and refine AI-generated code for efficiency and reliability.

Location and eligibility
Remote · Eligibility not specified
Engagement type
Contract
Compensation
Pay not listed
  • python
  • javascript
  • reactjs
Machine Learning & AIView job

micro1

Member of Technical Staff, Coding Agent Research & Evaluations

As a Member of Technical Staff, you will shape how coding agents are tested and improved. The work spans AI research, software engineering, and model evaluation: you will create benchmarks and datasets, study model failures, and build systems for experiments and evaluation. The role calls for at least three years of relevant technical experience and strong programming skills.

Location and eligibility
Remote · Eligibility not specified
Engagement type
Full-time
Compensation
$8–$9/hr
  • Coding Agent Evaluation & Benchmark Design
  • Software Engineering & Python
  • LLM & Agentic Systems
Machine Learning & AIView job

AfterQuery

Advanced Coding Expert

Create realistic, investigation-heavy coding challenges from a substantial private codebase you own. Your tasks will help evaluate and train advanced AI coding agents, so you’ll need to trace changes across large, multi-file projects and build containerized environments with tests that reflect careful engineering review.

Location and eligibility
Remote · Eligibility not specified
Engagement type
Not specified
Compensation
$85/hr
  • javascript
  • typescript
  • java
Software EngineeringView job

Turing

Senior Software Engineer, AI Evaluation and Coding Agents

Evaluate and improve AI coding models by reviewing generated code.

Location and eligibility
Remote · Eligibility not specified
Engagement type
Freelance
Compensation
$100–$150/hr
  • python
  • typescript
  • javascript
Machine Learning & AIView job