AIWorkJobs
SME Careers logoSME CareersPlatform listing

Healthcare Specialist – Rubric Design

Apply your healthcare specialty to build questions that test advanced AI systems and scoring rubrics that separate sound clinical reasoning from unsafe or incomplete answers. You will assess model responses, review…

Remote · United States onlyContract
Work arrangement
Remote
Eligibility
United States only
Engagement
Contract
Compensation
Up to $100/hr
Specialty
AI Training
Experience
Not specified
Interview language
English

About the role

Apply your healthcare specialty to build questions that test advanced AI systems and scoring rubrics that separate sound clinical reasoning from unsafe or incomplete answers. You will assess model responses, review colleagues’ rubrics, and create reference material for AI evaluation and training workflows. Applicants need a recognized healthcare qualification, at least three years of experience, and English proficiency at C1 or higher.

What you would do

  • Write specialty-specific questions that reveal gaps in AI-generated clinical responses.
  • Build scoring rubrics that spell out essential facts, reasoning, caveats, acceptable alternatives, and critical errors.
  • Separate core clinical criteria from supporting details when defining answer quality.
  • Review AI responses for factual accuracy, completeness, reasoning, safety, calibration, and relevance.
  • Flag unsupported assumptions, excessive certainty, missing considerations, and errors with potential clinical impact.
  • Review rubrics from other healthcare professionals for ambiguity, weak scoring criteria, redundancy, or omissions.
  • Prepare reference answers, explanations, and critiques to establish expert-level standards.
  • Produce questions, rubrics, evaluations, and reference-answer pairs for RLHF and SFT workflows.
  • Check written material for appropriate healthcare terminology and evidence-aware professional judgment.

Requirements

  • Hold a professional degree, license, credential, or recognized qualification in medicine, dentistry, pharmacy, nursing, or another healthcare discipline.
  • Have at least three years of professional or clinical experience.
  • Demonstrate expertise in a defined healthcare specialty or subspecialty.
  • Write and communicate in English at a professional level of C1 or higher.
  • Develop demanding clinical or professional questions that call for nuanced subject expertise.
  • Write detailed evaluation rubrics covering required content, reasoning, caveats, valid alternatives, and critical errors.
  • Distinguish minor gaps from mistakes that affect clinical quality or safety.
  • Assess AI-generated content and provide clear critiques of rubrics written by other healthcare professionals.
  • Produce consistent written work independently.

Preferred qualifications

The source marks these as preferred, not required.

  • Experience in medical education, examination writing, competency assessment, clinical guidelines, teaching, peer review, or quality assurance.
  • Prior work with AI evaluation, RLHF, SFT, data annotation, model benchmarking, or rubric-based scoring.

Who should apply

Healthcare professionals with a recognized qualification, three or more years of practice, and depth in a defined specialty should apply. Medical educators, exam writers, and clinicians who have assessed AI outputs or used rubric-based scoring bring relevant preferred experience.

How applying works on SME Careers

Create an account and submit the listing application with your CV. The official site does not publish the later screening steps.

  1. Sign up with LinkedIn or email.
  2. Open the listing and submit the application with your CV.
  3. Wait for the platform to contact you about any assessment or next step.

Paid steps: Not stated by the source

SME Careers does not publicly document its full screening process. Some roles may add an assessment; confirm with the platform.

Process checked Nov 19, 2026. AIWorkJobs does not submit applications or charge a fee. Read the full SME Careers process and its official source on the SME Careers page.

Skills mentioned by the source

  • healthcare
  • clinical pharmacology
  • clinical reasoning
  • medicine
  • dentistry
  • nursing
  • pharmacy
  • pathology
  • radiology
  • emergency medicine
  • surgery
  • medical education
  • rubric design
  • rubric writing
  • ai evaluation
  • expert review

Specialty areas

Specialties come from the source platform’s category taxonomy and are used to filter the job index.

Listing reference: healthcare-specialist-rubric-design-c26511. Source: SME Careers. If something here contradicts the source, the source controls. Report a correction.

Related roles

Other work on the index.

See more in this category

SME Careers

STEM & Technical Specialist – Rubric Design

Create advanced questions in your technical discipline, assess AI-generated solutions, and build scoring criteria that capture sound reasoning as well as correct results. The work centers on rubric design, partial-credit standards, and close review of technical details such as assumptions, formulas, units, and edge cases. Applicants need a relevant degree, at least three years of domain experience, and C1-level English.

Location and eligibility
Remote · United States only
Engagement type
Contract
Compensation
Up to $100/hr
  • ai evaluation
  • benchmarking
  • chemistry
Quality Assurance & TestingView job

AfterQuery

Healthcare Quality Specialist Expert

AfterQuery seeks healthcare quality professionals to create patient safety scenarios for AI evaluations and assess model responses. You’ll build cases from event reports through corrective action plans, explain the reasoning behind reference answers, and judge whether analyses meet methodological and regulatory standards. The work is remote and asynchronous, with a commitment of at least 10 hours per week and no fixed schedule.

Location and eligibility
Remote · Eligibility not specified
Engagement type
Not specified
Compensation
$100/hr
  • healthcare quality
  • patient safety
  • performance improvement
Health Informatics & Medical RecordsView job

Mercor

Medical Expert (M.D./D.O.)

Use your clinical judgment to help AI research teams assess model responses. Work may include creating clinical vignettes, grading diagnostic reasoning, and identifying recommendations that fail to account for a patient’s circumstances. This standing physician pool is not a specific opening; Mercor contacts candidates when a project matches their background, which may take a week or several months. Apply with a resume, complete an AI interview of about 20 minutes, and verify your license and background.

Location and eligibility
Remote · Eligibility not specified
Engagement type
Part-time
Compensation
$115–$150/hr
  • clinical reasoning
  • diagnostic reasoning
  • differential diagnosis
Healthcare & MedicalView job

Turing

Clinical & Health Specialist

Develop and evaluate complex clinical tasks and scenarios for accuracy.

Location and eligibility
Remote · Eligibility not specified
Engagement type
Contract
Compensation
Pay not listed
  • clinical reasoning
  • chart review
  • clinical documentation
Healthcare & MedicalView job