AIWorkJobs
Mercor logoMercorPlatform listing

Applied Engineer, Evaluations Role

Build and refine software engineering evaluations from completed pull requests, using Mercor’s in-house framework to assess frontier coding agents such as Claude Code and Codex. You’ll work with an internal research…

Remote · Eligibility not specifiedPart-time
Work arrangement
Remote
Eligibility
Eligibility not specified
Engagement
Part-time
Compensation
$100–$120/hr
Specialty
Machine Learning & AI

About the role

Build and refine software engineering evaluations from completed pull requests, using Mercor’s in-house framework to assess frontier coding agents such as Claude Code and Codex. You’ll work with an internal research team whose benchmarks compare frontier large language models, then take ownership of evaluation selection, development, and refinement after onboarding.

What you would do

  • Find repositories with enough substantial engineering work for challenging evaluations, then choose suitable completed pull requests.
  • Study each problem, its reference solution, and the repository’s architecture, tests, and conventions.
  • Write task prompts and create evaluations with automated tests, shell commands, and LLM grading prompts.
  • Check evaluations against reference solutions and deliberately flawed implementations. Identify missing checks, incorrect grades, and criteria that restrict valid solutions without cause.
  • Run evaluations across multiple models and harnesses, then determine whether failures stem from the agent’s solution, the environment, or the grading.
  • Confirm that each task reveals meaningful weaknesses in agent performance.
  • Refine evaluations through repeated testing and review, document findings and grading decisions, and address feedback.

Requirements

  • Strong programming fundamentals and practical experience with substantial, complex codebases.
  • Ability to read unfamiliar code, investigate subtle behavior, and judge whether different implementations solve the same problem correctly.
  • Experience writing meaningful tests, including edge-case and regression coverage.
  • Careful attention to detail and patience for repeated investigation and refinement.
  • Practical experience with AI coding tools and the judgment to verify their output and catch errors.
  • Confidence with Git, test runners, and CI tooling.
  • Clear written and spoken communication, with the ability to take ownership of tasks and raise questions when requirements lack clarity.
  • After onboarding, ability to select tasks, develop evaluations, and iterate without step-by-step direction.

Preferred qualifications

The source marks these as preferred, not required.

  • Familiarity with a particular programming language or ecosystem.

Who should apply

Software engineers who can make sound judgments in unfamiliar codebases and build tests that distinguish correct solutions from flawed ones should consider this role. Experience can come from open-source, private, or enterprise repositories; public contributions and work in a popular ecosystem matter less than the ability to learn the codebase.

How applying works on Mercor

Create a profile, complete the role checklist and AI interview, then verify your identity before a project offer.

  1. Create an account and confirm your résumé.
  2. Complete the role checklist, written assessment, and AI interview where the listing requires them.
  3. Submit the application and wait for review or matching.
  4. Complete a work trial and identity verification before onboarding.

Typical timeline: The AI interview is about 20 minutes; work trials are often 4–6 hours; identity checks are usually minutes.

Paid steps: The work trial is paid.

Assessment and interview requirements vary by role. Check the checklist inside the specific listing.

Process checked Nov 19, 2026. AIWorkJobs does not submit applications or charge a fee. Read the full Mercor process and its official source on the Mercor page.

Skills mentioned by the source

  • typescript
  • javascript
  • python
  • java
  • kotlin
  • go
  • ruby
  • php
  • c++
  • rust
  • git
  • automated testing
  • regression testing
  • ci
  • shell scripting
  • llm evaluation

Specialty areas

Specialties come from the source platform’s category taxonomy and are used to filter the job index.

Listing reference: applied-engineer-evaluations-role-NNEa6l. Source: Mercor. If something here contradicts the source, the source controls. Report a correction.

Related roles

Other work on the index.

See more in this category

Turing

Senior Software Engineer - AI Evaluation / Coding Agents

Evaluate and improve AI coding models by reviewing generated code.

Location and eligibility
Remote · Eligibility not specified
Engagement type
Contract
Compensation
$100–$150/hr
  • python
  • typescript
  • javascript
Machine Learning & AIView job

micro1

Member of Technical Staff, Coding Agent Research & Evaluations

Shape how frontier coding agents are tested and improved. You will create evaluation methods, datasets, and tools, then study model performance across software engineering tasks. The work combines Python and software engineering, LLM and agent systems, benchmark design, and model failure analysis. You will also share research findings with technical and client audiences.

Location and eligibility
Remote · Eligibility not specified
Engagement type
Full-time
Compensation
$8–$9/hr
  • Coding Agent Evaluation & Benchmark Design
  • Software Engineering & Python
  • LLM & Agentic Systems
Machine Learning & AIView job

Turing

Senior Software Engineer – LLM Evaluation

Evaluate and refine AI-generated code for efficiency and reliability.

Location and eligibility
Remote · Eligibility not specified
Engagement type
Contract
Compensation
Pay not listed
  • python
  • javascript
  • reactjs
Machine Learning & AIView job

Turing

Senior Python Engineer – LLM Evaluation (US-based)

Create and evaluate datasets to enhance large language model performance.

Location and eligibility
Remote · United States only
Engagement type
Contract
Compensation
Pay not listed
  • python
  • c++
  • c
Machine Learning & AIView job