AIWorkJobs
Braintrust logoLeading AI LabListed on Braintrust

Principal Coding Annotator / LLM Evaluation Engineer

Bring hands-on software engineering judgment to the evaluation of large language models. You’ll create coding benchmarks, assess model responses, and investigate security-related tasks and reasoning failures. The…

On-site · Eligible in 10 countriesContract
Work arrangement
On-site
Eligibility
Eligible in 10 countries
Engagement
Contract
Compensation
$75–$90/hr
Specialty
Machine Learning & AI
Experience
10+ years

About the role

Bring hands-on software engineering judgment to the evaluation of large language models. You’ll create coding benchmarks, assess model responses, and investigate security-related tasks and reasoning failures. The contract starts at six months, with potential for a longer engagement. Paris or London candidates are preferred; strong candidates elsewhere in Europe may work remotely.

What you would do

  • Assess coding tasks involving software vulnerabilities, exploit verification, and security patches.
  • Write challenging coding prompts and reference solutions for benchmark-style problems, including SWE-Bench-like tasks.
  • Evaluate LLM responses to code generation, refactoring, debugging, and implementation challenges.
  • Record model failures, edge cases, and gaps in reasoning.
  • Compare private Mistral-based LLMs with leading external models.
  • Build or configure coding environments for evaluation and reinforcement learning workflows.
  • Apply detailed annotation and evaluation guidelines consistently.

Requirements

  • At least 5 years of professional software development experience.
  • Strong Python skills.
  • Hands-on experience with vulnerability research, exploit reproduction or verification, or implementing, backporting, or validating security patches.
  • Ability to use structured evaluation criteria and write clear technical feedback.
  • Fluent written and spoken English.

Preferred qualifications

The source marks these as preferred, not required.

  • Knowledge of at least one programming language beyond Python.
  • Experience in professional code review, coding annotation, LLM or code evaluation, or benchmark design.
  • Team leadership or mentoring experience.

Who should apply

Experienced software engineers with at least five years in the field and strong Python skills should consider this role, especially those with hands-on security or vulnerability work. Candidates must communicate fluently in English and be able to assess code against clear technical criteria.

How applying works on Braintrust

Create a Braintrust profile, complete the AI skills interview, then browse and accept tasks from the dashboard.

  1. Create and complete a Braintrust profile.
  2. Complete the AI skills interview used for expertise and pay-level matching.
  3. Browse the dashboard for available tasks.
  4. Accept a task and complete the work.

Typical timeline: A new member profile takes about 20 minutes.

Paid steps: Not stated by the source

Individual client projects can add their own interview, assessment, or identity check.

Process checked Nov 19, 2026. AIWorkJobs does not submit applications or charge a fee. Read the full Braintrust process and its official source on the Braintrust page.

Skills mentioned by the source

  • software engineering
  • python
  • data annotation
  • llm evaluation
  • code generation
  • code review
  • benchmark design
  • vulnerability research
  • exploit verification
  • security patches
  • reinforcement learning
  • swe-bench

Specialty areas

Specialties come from the source platform’s category taxonomy and are used to filter the job index.

Listing reference: principal-coding-annotator-llm-evaluation-engineer-16729. Source: Braintrust. If something here contradicts the source, the source controls. Report a correction.

Related roles

Other work on the index.

See more in this category

Turing

Senior Python Engineer – LLM Evaluation (US-based)

Create and evaluate datasets to enhance large language model performance.

Location and eligibility
Remote · United States only
Engagement type
Contract
Compensation
Pay not listed
  • python
  • c++
  • c
Machine Learning & AIView job

Turing

Senior Python Engineer – LLM Evaluation (US-based)

Create and refine Python solutions for training large language models.

Location and eligibility
Remote · United States only
Engagement type
Contract
Compensation
Pay not listed
  • python
  • javascript
  • reactjs
Machine Learning & AIView job