AIWorkJobs
SME Careers logoSME CareersPlatform listing

Legal & Compliance Specialist – Rubric Design

Create demanding legal and compliance scenarios, then build rubrics that specify the issues, standards, reasoning, exceptions, and risks a sound answer should address. You will assess AI-generated analysis for…

Remote · United States onlyContract
Work arrangement
Remote
Eligibility
United States only
Engagement
Contract
Compensation
Up to $120/hr
Specialty
Legal
Experience
Not specified
Interview language
English

About the role

Create demanding legal and compliance scenarios, then build rubrics that specify the issues, standards, reasoning, exceptions, and risks a sound answer should address. You will assess AI-generated analysis for accuracy, completeness, and sound judgment, with subject matter drawn from your area of expertise, such as contracts, privacy, or financial regulation.

What you would do

  • Write realistic legal and compliance scenarios that test issue spotting, interpretation, and professional judgment.
  • Build scoring rubrics that define relevant issues, standards, reasoning steps, exceptions, risks, and critical errors.
  • Evaluate AI-generated analysis for accuracy, completeness, relevance, and quality of reasoning.
  • Check whether model responses identify material legal or regulatory issues and apply suitable rules or standards.
  • Flag unsupported conclusions, faulty assumptions, missed exceptions, and weak treatment of risk.
  • Review other experts’ rubrics for ambiguity, omissions, redundancy, subjectivity, and inaccurate expectations.
  • Prepare reference analyses, answer frameworks, critiques, and gold-standard examples.
  • Create questions, evaluations, preference data, and reference-answer pairs for RL and SFT workflows.
  • Separate factual analysis, legal interpretation, professional judgment, and uncertainty in evaluation content.

Requirements

  • Hold a JD, LLB, LLM, law degree, compliance qualification, or equivalent professional background relevant to your specialty.
  • Bring at least three years of professional experience in law, compliance, legal research, policy, risk, privacy, financial compliance, contracts, or a related area.
  • Show expertise in a defined legal, regulatory, or compliance domain.
  • Demonstrate C1 or higher professional English proficiency.
  • Create challenging scenarios that test issue spotting, interpretation, analysis, and judgment.
  • Write detailed rubrics that distinguish required issues, applicable standards, exceptions, risks, and critical omissions.
  • Recognize valid differences in legal interpretation and distinguish them from objectively deficient analysis.
  • Review expert-authored rubrics and assess terminology, jurisdiction, authority, factual assumptions, materiality, and conclusions.

Preferred qualifications

The source marks these as preferred, not required.

  • Professional legal qualification or bar admission for attorney-focused work.
  • Background in legal education, examination writing, peer review, legal editing, quality assurance, policy drafting, compliance testing, or supervisory review.
  • Experience with AI evaluation, legal AI, data annotation, RLHF, model benchmarking, or rubric-based scoring.

Who should apply

Legal and compliance professionals with three or more years of relevant experience and expertise in a defined practice area should apply. Attorneys and other specialists who can turn nuanced analysis into clear scoring criteria, and who have experience evaluating AI or expert work, may be especially suited to the role.

How applying works on SME Careers

Create an account and submit the listing application with your CV. The official site does not publish the later screening steps.

  1. Sign up with LinkedIn or email.
  2. Open the listing and submit the application with your CV.
  3. Wait for the platform to contact you about any assessment or next step.

Paid steps: Not stated by the source

SME Careers does not publicly document its full screening process. Some roles may add an assessment; confirm with the platform.

Process checked Nov 19, 2026. AIWorkJobs does not submit applications or charge a fee. Read the full SME Careers process and its official source on the SME Careers page.

Skills mentioned by the source

  • ai evaluation
  • compliance
  • contract law
  • corporate law
  • english
  • expert review
  • financial regulation
  • issue spotting
  • legal analysis
  • legal reasoning
  • model evaluation
  • privacy
  • regulatory compliance
  • risk assessment
  • rubric design
  • rubric writing

Specialty areas

Specialties come from the source platform’s category taxonomy and are used to filter the job index.

Listing reference: legal-compliance-specialist-rubric-design-c26513. Source: SME Careers. If something here contradicts the source, the source controls. Report a correction.

Related roles

Other work on the index.

See more in this category

SME Careers

Finance & Accounting Specialist – Rubric Design

Use your finance or accounting expertise to create realistic professional problems and assess AI-generated analyses. You will build scoring rubrics that capture sound calculations, assumptions, methods, and judgment, then review outputs and peer criteria for technical accuracy and completeness across areas such as valuation, reporting, audit, and investment analysis.

Location and eligibility
Remote · United States only
Engagement type
Contract
Compensation
Up to $100/hr
  • accounting
  • finance
  • financial analysis
Finance & AccountingView job

AfterQuery

Compliance Officer Expert

Create and assess regulatory compliance challenges that help train AI models. You will write scenarios on topics such as regulatory interpretation, risk assessment, audits, investigations, and ethics, then review model answers for accuracy and alignment with compliance standards. The work draws on practical compliance expertise and takes place remotely with a flexible, asynchronous schedule.

Location and eligibility
Remote · Eligibility not specified
Engagement type
Not specified
Compensation
$75/hr
  • regulatory compliance
  • regulatory frameworks
  • risk assessment
Machine Learning & AIView job

Flash · via Mercor

AI Evaluation Rubric Auditor

Audit complete AI-agent shopping and delivery tasks, including the prompt and grading rubric. You will judge whether each rubric failure was fair by checking the available information, the prompt’s requirements, the treatment of valid answers, and the rubric’s structure. You will also give reviewers specific feedback on prompt and rubric issues.

Location and eligibility
Remote · Eligibility not specified
Engagement type
Hourly
Compensation
$60–$85/hr
  • llm evaluation
  • rubric design
  • rubric auditing
AI evaluationView job

Turing

Securities & Financial Regulation Expert

Review AI responses to U.S. securities law scenarios for accuracy.

Location and eligibility
Remote · Eligibility not specified
Engagement type
Freelance
Compensation
Pay not listed
  • securities regulation
  • securities act
  • securities exchange act
LegalView job