AIWorkJobs
Turing logoTuringPlatform listing

Senior Software Engineer – LLM Evaluation & Repository Validation

Analyze GitHub issues and evaluate LLM performance on real code.

Remote · Eligibility not specifiedContract
Work arrangement
Remote
Eligibility
Eligibility not specified
Engagement
Contract
Compensation
Pay not listed
Specialty
Software Engineering

About the role

Help build LLM evaluation and training datasets from public repository histories. You’ll assess how models handle real software engineering tasks, including bug fixes, and help expand task coverage across languages and difficulty levels. The work combines hands-on repository setup, issue triage, and test-quality review, with some overlap with PST required.

What you would do

  • Review and triage issues across trending open-source libraries.
  • Set up repositories, including Docker configuration and development environments.
  • Assess unit test coverage and test quality.
  • Run and modify codebases to evaluate LLM performance on bug-fixing tasks.
  • Work with researchers to select repositories and issues that challenge LLMs.
  • Help lead junior engineers on project work when needed.

Requirements

  • Bring strong experience in at least one of these languages: Python, JavaScript, Java, Go, Rust, C, C++, C#, or Ruby.
  • Have experience with well-maintained public repositories that have 500+ stars.
  • Use Git and Docker, and understand basic software pipeline setup.
  • Navigate complex codebases and run, modify, and test real-world projects on a local machine.
  • Bring senior or tech lead-level software engineering experience.
  • Commit 20 hours per week and provide some overlap with PST.
  • Accept a three-month contractor assignment with no medical benefits or paid leave.

Preferred qualifications

The source marks these as preferred, not required.

  • Experience contributing to or evaluating open-source projects.
  • Prior involvement in LLM research or evaluation projects.
  • Experience building or testing developer tools or automation agents.

Who should apply

Senior software engineers with hands-on experience in established public repositories and strong skills in at least one listed programming language should consider this assignment. The role suits candidates who can set up and test real codebases and commit 20 hours each week with some PST overlap.

How applying works on Turing

Create a profile, finish any required proctored assessments, then complete screening or a role interview.

  1. Apply through work.turing.com and complete your profile.
  2. Finish the required proctored assessments for the role.
  3. Complete shortlisting, screening, or a technical interview.
  4. Confirm rate, availability, and eligibility before an offer.

Typical timeline: Review takes 48 hours to six weeks after all required steps; consideration replies are usually within 12 hours.

Paid steps: Not stated by the source

Not every role requires tests. Turing does not review an application until required assessments are complete.

Process checked Nov 19, 2026. AIWorkJobs does not submit applications or charge a fee. Read the full Turing process and its official source on the Turing page.

Skills mentioned by the source

  • python
  • javascript
  • java
  • go
  • rust
  • c
  • c++
  • c#
  • ruby
  • git
  • docker
  • software pipelines
  • github
  • unit testing
  • llm evaluation
  • repository validation

Specialty areas

Specialties come from the source platform’s category taxonomy and are used to filter the job index.

Listing reference: senior-software-engineer-llm-evaluation-repository-validation-34123. Source: Turing. If something here contradicts the source, the source controls. Report a correction.

Related roles

Other work on the index.

See more in this category