AIWorkJobs
Mercor logoMercorPlatform listing

SWE-Bench Task Auditor

Review repository-level software engineering benchmarks used to train and evaluate a frontier AI lab’s models. You’ll examine task quality, reference patches, test harnesses, Docker isolation, and grading integrity…

Remote · United States onlyHourly
Work arrangement
Remote
Eligibility
United States only
Engagement
Hourly
Compensation
$70–$90/hr
Specialty
Quality Assurance & Testing
Weekly commitment
40 hours per week

About the role

Review repository-level software engineering benchmarks used to train and evaluate a frontier AI lab’s models. You’ll examine task quality, reference patches, test harnesses, Docker isolation, and grading integrity, then write clear feedback against a rubric. The role calls for professional software engineering experience and open-source contribution or maintainer work. The listing is limited to the United States.

What you would do

  • Assess the quality and correctness of repository-level benchmark tasks.
  • Check reference patches against the task they aim to solve.
  • Audit test harnesses, test runners, and Docker isolation.
  • Identify answer leakage and reward-hacking risks.
  • Evaluate whether benchmark tasks and their grading are reproducible and sound.
  • Write clear, rubric-based feedback on your findings.

Requirements

  • At least 3 years of professional software engineering experience.
  • A record of real open-source contributions, such as merged pull requests, or experience as a committer or maintainer.
  • Strong ability to audit reference patches, test runners, and Docker isolation.
  • Ability to detect answer leakage and reward hacking.
  • Fluency in Python and at least one of Java, Go, TypeScript, or C++.
  • Must be based in the United States.

Preferred qualifications

The source marks these as preferred, not required.

  • Familiarity with SWE-Bench Verified or similar repository benchmarks.
  • Maintainer history on a major Python open-source project, such as Django, Flask, scikit-learn, sympy, or pytest.
  • Prior code review or task-grading experience.

Who should apply

Software engineers with at least three years of professional experience and meaningful open-source contributions should consider this role. Candidates need Python plus another listed language, and the ability to judge benchmark patches, tests, and grading integrity.

How applying works on Mercor

Create a profile, complete the role checklist and AI interview, then verify your identity before a project offer.

  1. Create an account and confirm your résumé.
  2. Complete the role checklist, written assessment, and AI interview where the listing requires them.
  3. Submit the application and wait for review or matching.
  4. Complete a work trial and identity verification before onboarding.

Typical timeline: The AI interview is about 20 minutes; work trials are often 4–6 hours; identity checks are usually minutes.

Paid steps: The work trial is paid.

Assessment and interview requirements vary by role. Check the checklist inside the specific listing.

Process checked Nov 19, 2026. AIWorkJobs does not submit applications or charge a fee. Read the full Mercor process and its official source on the Mercor page.

Skills mentioned by the source

  • python
  • java
  • go
  • typescript
  • c++
  • swe-bench
  • software engineering
  • open source
  • code review
  • test harnesses
  • docker
  • benchmark evaluation

Specialty areas

Specialties come from the source platform’s category taxonomy and are used to filter the job index.

Listing reference: swe-bench-task-auditor-ZAV5rw. Source: Mercor. If something here contradicts the source, the source controls. Report a correction.

Related roles

Other work on the index.

See more in this category

Turing

SWE Bench – Data Engineer/Data Scientist

Build and validate data pipelines for AI performance evaluation projects.

Location and eligibility
Remote · Eligibility not specified
Engagement type
Contract
Compensation
Pay not listed
  • python
  • data engineering
  • data science
Data EngineeringView job

Outlier · via Braintrust

Senior Systems & Hardware Experts - AI Benchmarking

Use your systems or hardware engineering experience to create difficult, clearly specified challenges for AI agents to solve in a Linux terminal. The project is task-based and expected to last a few weeks. Tasks average six to seven hours, with early submissions potentially taking longer. Earn up to $1,350 per task, with higher rates possible based on task quality and submission volume. Onboarding requires two courses, a graded assessment of about 90 minutes, and a live webinar.

Location and eligibility
Remote · Eligible in 25 countries
Engagement type
Contract
Compensation
$1,350 fixed
  • systems engineering
  • hardware engineering
  • linux
Hardware & Embedded SystemsView job

Meridial by Invisible

Technical Task Auditor

Audit technical tasks used to train and evaluate AI systems as an independent contractor. You will focus on one specialty that matches your experience, such as GPU kernels, Kubernetes, machine learning, or application security. Your reviews will check task quality, reproducibility, tests, and evaluation criteria, and you will provide actionable feedback on any problems.

Location and eligibility
Remote · Eligibility not specified
Engagement type
Not specified
Compensation
$60/hr
  • gpu kernels
  • aws trainium
  • nki
Quality Assurance & TestingView job

Mercor

Kubernetes Task Auditor

Review Kubernetes tasks used to train and evaluate models for a frontier AI lab. You will judge cluster-operation scenarios, check manifest accuracy and production readiness, and assess troubleshooting for failure modes. Your written feedback must follow clear rubrics. The role calls for at least three years of production Kubernetes experience and hands-on incident debugging.

Location and eligibility
Remote · United States only
Engagement type
Hourly
Compensation
$70–$90/hr
  • kubernetes
  • eks
  • gke
Cloud & DevOpsView job