AIWorkJobs
Mercor logoFlashListed on Mercor

AI Evaluation Rubric Auditor

Audit AI agents’ work on realistic shopping and delivery tasks by reviewing the prompt, grading rubric, and full agent transcript. You’ll judge whether the rubric treats failures fairly, follows the guidelines, and…

Remote · Eligibility not specifiedHourly
Work arrangement
Remote
Eligibility
Eligibility not specified
Engagement
Hourly
Compensation
$60–$85/hr
Specialty
AI Training
Weekly commitment
15 hours per week

About the role

Audit AI agents’ work on realistic shopping and delivery tasks by reviewing the prompt, grading rubric, and full agent transcript. You’ll judge whether the rubric treats failures fairly, follows the guidelines, and matches the prompt, then give reviewers specific feedback on prompt and rubric issues. Applicants need substantial experience writing or reviewing rubrics for LLM evaluations and must list those projects.

What you would do

  • Review complete tasks, including the prompt and grading rubric.
  • Check whether the agent had access to the information needed to answer.
  • Assess whether the rubric grades the prompt’s actual requirements.
  • Identify valid answers that the rubric would mark wrong.
  • Check rubric structure against the applicable guidelines.
  • Trace long agent transcripts to find where an answer went wrong.
  • Give reviewers feedback on prompt and rubric issues.

Requirements

  • Substantial experience writing or reviewing rubrics for LLM evaluations.
  • List relevant rubric projects in your application.
  • Experience giving feedback to writers or reviewers.
  • Consistent availability.
  • Ability to read long agent transcripts and identify where an agent went wrong.
  • Precise writing and a habit of checking the data rather than trusting the reference answer.

Who should apply

Candidates with substantial LLM evaluation rubric experience who can audit complete shopping and delivery tasks, trace long agent transcripts, and give clear feedback to reviewers. Include your relevant projects in your application.

How applying works on Mercor

Create a profile, complete the role checklist and AI interview, then verify your identity before a project offer.

  1. Create an account and confirm your résumé.
  2. Complete the role checklist, written assessment, and AI interview where the listing requires them.
  3. Submit the application and wait for review or matching.
  4. Complete a work trial and identity verification before onboarding.

Typical timeline: The AI interview is about 20 minutes; work trials are often 4–6 hours; identity checks are usually minutes.

Paid steps: The work trial is paid.

Assessment and interview requirements vary by role. Check the checklist inside the specific listing.

Process checked Nov 19, 2026. AIWorkJobs does not submit applications or charge a fee. Read the full Mercor process and its official source on the Mercor page.

Skills mentioned by the source

  • llm evaluation
  • evaluation rubrics
  • rubric auditing
  • prompt evaluation
  • agent transcript analysis
  • data verification
  • reviewer feedback
  • grading criteria

Listing reference: ai-evaluation-rubric-auditor-9DVrUb. Source: Mercor. If something here contradicts the source, the source controls. Report a correction.

Related roles

Other work on the index.

See more in this category

SME Careers

Legal & Compliance Specialist – Rubric Design

Create demanding legal and compliance scenarios, then build rubrics that specify the issues, standards, reasoning, exceptions, and risks a sound answer should address. You will assess AI-generated analysis for accuracy, completeness, and sound judgment, with subject matter drawn from your area of expertise, such as contracts, privacy, or financial regulation.

Location and eligibility
Remote · United States only
Engagement type
Contract
Compensation
Up to $120/hr
  • ai evaluation
  • compliance
  • contract law
LegalView job

Turing

Senior Software Engineer, AI Evaluation and Coding Agents

Evaluate and improve AI coding models by reviewing generated code.

Location and eligibility
Remote · Eligibility not specified
Engagement type
Freelance
Compensation
$100–$150/hr
  • python
  • typescript
  • javascript
Machine Learning & AIView job

micro1

AI Agent Evaluator

Test two AI agents on your phone by asking them to handle everyday errands, then review what happened from a user’s perspective. Each run takes about 1.5 hours, including recording evidence and scoring results. You’ll need an active personal account with a popular AI service, such as ChatGPT, and a smartphone with screen recording. After presentation to the hiring manager, onboarding typically takes 24–48 hours.

Location and eligibility
Remote · United States only
Engagement type
Contract
Compensation
$25–$45/hr
  • ChatGPT
  • Muse
  • AI agents
Quality Assurance & TestingView job

AfterQuery

Legal Experts

Help train and assess AI models that reason about the law. You may write challenging legal prompts, review answers for accuracy and sound reasoning, catch issues such as fabricated citations or misapplied rules, draft model responses with explanations, and build rubrics for grading answer quality. The application requests Bar Admission credentials.

Location and eligibility
Remote · Eligibility not specified
Engagement type
Not specified
Compensation
$125/hr
  • legal reasoning
  • legal prompts
  • legal accuracy
LegalView job