Find Jobs in AI All opportunities
Open for applicationsAI training

AI Evaluation Rubric Auditor

Open role
Pay
$60–$85 / hour
Location
Remote
Experience level
Experienced

A CLOSER LOOK

Explore the opportunity

01 / THE DETAILS

Role overview

Get to know the opportunity

Audit reviewer decisions on tests that measure how AI agents handle online shopping and delivery tasks.

02 / THE DETAILS

What you’ll be doing

Your day-to-day contribution
  • Review the full task, its scoring criteria and the reviewer’s assessment.
  • Check whether the agent had enough information, whether the scoring matches the prompt and whether a correct answer could be rejected.
  • Give reviewers specific feedback on ambiguous instructions, unfair scoring and rubric structure.
03 / THE DETAILS

Who this could suit

See where your experience could fit

Experienced language-model evaluators who can review other people’s assessments and explain how to improve them.

04 / THE DETAILS

What you’ll need

Experience, skills and essential requirements
  • Substantial experience creating or reviewing language-model evaluation rubrics, experience giving feedback to reviewers or writers, and consistent availability.
  • You must be able to trace issues in long agent transcripts, check the underlying evidence and write precisely.
  • Be ready to name relevant evaluation projects.

You don’t have to match every requirement exactly. We welcome applicants with different backgrounds, levels of experience and transferable skills.

05 / THE DETAILS

Pay and working arrangements

Rates, commitment and working arrangements

The listing states $60–$85 per hour and 15 hours per week for remote work. It does not name eligible countries or specify a project duration or contract classification.

06 / THE DETAILS

What happens next

How to move forward

Apply through Find Jobs in AI with your profile and CV. We’ll review your application and, if you’re a good fit, we’ll be in touch about the next steps.

Practice for this role

Check an AI answer against a source

Notice: “Returns are accepted within 30 days with a receipt. Opened software is excluded.” AI answer: “You can return any item within 45 days without a receipt.”

Identify the errors and write a corrected answer using the notice.

Sign in for private feedback or work through the example below.

See the rubric and example answer
  • Time limit · Correct 45 days to 30 days.
  • Receipt requirement · State that a receipt is required.
  • Exception · State that opened software is excluded.

Returns are accepted within 30 days with a receipt. Opened software is excluded.

Application checklist
Requirements from the listing

Substantial experience creating or reviewing language-model evaluation rubrics, experience giving feedback to reviewers or writers, and consistent availability. You must be able to trace issues in long agent transcripts, check the underlying evidence and write precisely. Be ready to name relevant evaluation projects.

Sign in to check your saved CV.

Pay$60–$85 / hour
Continue to apply