01 / THE DETAILSRole overview
Get to know the opportunity
Audit reviewer decisions on tests that measure how AI agents handle online shopping and delivery tasks.
02 / THE DETAILSWhat you’ll be doing
Your day-to-day contribution
- Review the full task, its scoring criteria and the reviewer’s assessment.
- Check whether the agent had enough information, whether the scoring matches the prompt and whether a correct answer could be rejected.
- Give reviewers specific feedback on ambiguous instructions, unfair scoring and rubric structure.
03 / THE DETAILSWho this could suit
See where your experience could fit
Experienced language-model evaluators who can review other people’s assessments and explain how to improve them.
04 / THE DETAILSWhat you’ll need
Experience, skills and essential requirements
- Substantial experience creating or reviewing language-model evaluation rubrics, experience giving feedback to reviewers or writers, and consistent availability.
- You must be able to trace issues in long agent transcripts, check the underlying evidence and write precisely.
- Be ready to name relevant evaluation projects.
You don’t have to match every requirement exactly. We welcome applicants with different backgrounds, levels of experience and transferable skills.
05 / THE DETAILSPay and working arrangements
Rates, commitment and working arrangements
The listing states $60–$85 per hour and 15 hours per week for remote work. It does not name eligible countries or specify a project duration or contract classification.
06 / THE DETAILSWhat happens next
How to move forward
Apply through Find Jobs in AI with your profile and CV. We’ll review your application and, if you’re a good fit, we’ll be in touch about the next steps.