AIWorkJobs
Turing logoTuringPlatform listing

Top GitHub Contributors – Agentic Review Project

Design and analyze challenging benchmark tasks to enhance AI reasoning.

Remote · Eligibility not specifiedContract
Work arrangement
Remote
Eligibility
Eligibility not specified
Engagement
Contract
Compensation
Pay not listed
Specialty
AI Training

About the role

Design and validate challenging benchmark tasks that test a frontier model’s reasoning and problem-solving. You will build task materials in an Agentic development environment with Python, then assess model results and trace failures to logical reasoning gaps.

What you would do

  • Create real-world benchmark problems that target specific capability loss failures in a leading model.
  • Prepare each task in an Agentic development environment, including instructions, an overview, a golden solution, datasets, Python libraries, and metadata.
  • Write a test notebook with unit tests that solutions must pass.
  • Evaluate cross-model performance on the benchmark tasks.
  • Find tasks where the target model fails to pass every test and classify those cases as logical reasoning failures.
  • Review agent trajectories and identify core capability loss patterns.

Requirements

  • Be a highly skilled GitHub contributor.
  • Have strong expertise in at least one of these areas: data science, machine learning, coding, or quantitative work in frontier STEM.
  • Commit to 30 or more hours per week on weekdays.
  • Complete the required AI video interview as part of the application process.

Who should apply

Experienced GitHub contributors with strong skills in data science, machine learning, coding, or quantitative STEM work should consider this project. Candidates must have capacity to contribute at least 30 hours each week on weekdays and complete an AI video interview.

How applying works on Turing

Create a profile, finish any required proctored assessments, then complete screening or a role interview.

  1. Apply through work.turing.com and complete your profile.
  2. Finish the required proctored assessments for the role.
  3. Complete shortlisting, screening, or a technical interview.
  4. Confirm rate, availability, and eligibility before an offer.

Typical timeline: Review takes 48 hours to six weeks after all required steps; consideration replies are usually within 12 hours.

Paid steps: Not stated by the source

Not every role requires tests. Turing does not review an application until required assessments are complete.

Process checked Nov 19, 2026. AIWorkJobs does not submit applications or charge a fee. Read the full Turing process and its official source on the Turing page.

Skills mentioned by the source

  • rust
  • python
  • github
  • data science
  • machine learning
  • coding
  • quantitative analysis
  • stem
  • benchmark design
  • unit testing
  • model evaluation
  • logical reasoning
  • agentic systems

Listing reference: top-github-contributors-agentic-review-project-64992. Source: Turing. If something here contradicts the source, the source controls. Report a correction.

Related roles

Other work on the index.

See more in this category

Turing

AI Benchmark & Red-Teaming Specialist

Research and build benchmark tasks to expose AI model weaknesses.

Location and eligibility
Remote · Eligibility not specified
Engagement type
Contract
Compensation
Pay not listed
  • ai benchmarking
  • red teaming
  • large language models
Machine Learning & AIView job

Outlier · via Braintrust

Senior Coding, ML & Security Experts - AI Benchmarking

Create difficult benchmark tasks from your professional experience in coding, machine learning, or cybersecurity. Each task must run in a Linux terminal and challenge advanced AI agents: three leading models get three attempts each, and they should fail most attempts. The project should run for the next few weeks. Onboarding requires two courses, a graded assessment of about 90 minutes, and a live webinar before tasking. Pay reaches $1,350 per task, with higher rates possible based on task quality and submission volume; tasks average six to seven hours, and your first tasks may take longer.

Location and eligibility
Remote · Eligible in 25 countries
Engagement type
Contract
Compensation
$1,350 fixed
  • coding
  • machine learning
  • cybersecurity
Machine Learning & AIView job

Mercor

SWE-Bench Task Auditor

Review repository-level software engineering benchmarks used to train and evaluate a frontier AI lab’s models. You’ll examine task quality, reference patches, test harnesses, Docker isolation, and grading integrity, then write clear feedback against a rubric. The role calls for professional software engineering experience and open-source contribution or maintainer work. The listing is limited to the United States.

Location and eligibility
Remote · United States only
Engagement type
Hourly
Compensation
$70–$90/hr
  • python
  • java
  • go
Quality Assurance & TestingView job

AfterQuery

Advanced Coding Expert

Create realistic, investigation-heavy coding challenges from a substantial private codebase you own. Your tasks will help evaluate and train advanced AI coding agents, so you’ll need to trace changes across large, multi-file projects and build containerized environments with tests that reflect careful engineering review.

Location and eligibility
Remote · Eligibility not specified
Engagement type
Not specified
Compensation
$85/hr
  • javascript
  • typescript
  • java
Software EngineeringView job