FIND JOBS IN AI GUIDE
What is a rubric in AI? Definition and example
A rubric is a set of criteria used to judge a piece of work. In AI evaluation, each criterion says what a response must get right and what evidence earns a score. Reviewers apply the same criteria to the same answer so the decision is more consistent and easier to explain.
By Find Jobs in AI · Updated
Rubric example: score an AI answer
Use only this fictional reference. The task is to state the workshop day, start time and laptop requirement.
Reference“The workshop is on Wednesday at 10:00. It lasts 45 minutes. Participants do not need to bring a laptop.”
“The workshop is on Wednesday at 10:00. Bring your laptop.”
| Criterion | What earns the point | Points |
|---|---|---|
| 1. Workshop day | States that the workshop is on Wednesday. | 0 or 1 |
| 2. Start time | States that it starts at 10:00. | 0 or 1 |
| 3. Laptop requirement | States that participants do not need to bring a laptop. | 0 or 1 |
Try scoring all three criteria before checking the explanation. The answer can pass one criterion and fail another.
Show the score and feedback
Score: 2 out of 3. The day and time each earn one point. The laptop instruction earns zero because it contradicts the reference.
A useful correction is: “Replace the instruction to bring a laptop with ‘You do not need to bring a laptop.’ The reference explicitly says one is not required.”
The duration earns no extra point because the task did not ask for it. These equal weights belong to this practice exercise; a real project sets its own criteria, weights and pass threshold.

Why do rubrics matter?
If two reviewers read the same answer and reach different scores, the problem may be an unclear standard rather than the answer itself. A rubric tells them which facts, behaviours or omissions matter and how to recognise them.
Rubrics can also make feedback more useful. Instead of “not good enough”, a reviewer can identify exactly which criterion was missed.
What do atomic and self-contained mean?
An atomic criterion asks one question. If an answer can satisfy half of the sentence but fail the other half, split the sentence into two criteria.
A self-contained criterion carries enough context to be judged without guessing the intended answer. It names the expected fact or behaviour, including the relevant task conditions where necessary.
Rubric, checklist or rating scale: what is the difference?
A checklist records whether an item is present or a step is complete. A rating scale gives levels such as 1 to 5. A rubric explains what the scores mean for the task, so the reviewer can connect a decision to observable evidence. A rubric can use a checklist-style pass/fail decision for each criterion, as the example above does.
For a scored 1-to-5 criterion, define the levels before reviewing the answers. A score of 3 should have an explanation another reviewer can apply; it should not mean only that the response feels average.
A simple rubric-writing template
For each criterion, write: what is being checked; the expected answer or behaviour; what evidence establishes a pass; and how many points it is worth. Add a separate route for a case that cannot be judged from the permitted material.
For example: “Laptop guidance — 1 point if the answer says a laptop is not required; 0 points if it says to bring one or omits the requirement.” This is grounded in the fictional reference above. If a different task never states the laptop rule, do not import it from this example.
Start with evidence, then write the criterion
Read the prompt and reference material first. Decide what a correct answer must include and what would be an actual error. Write a short pass condition for each distinct item, then check whether a different reviewer could use it without extra explanation.
Atomic means one decision per criterion
Do not combine several requirements with “and” if a response could meet one but miss another. Split “identifies the issue and explains the exception” into two criteria when each is independently important. This makes partial credit and error analysis clearer.
Atomic does not mean reducing a judgment to meaningless fragments. A criterion should still test a useful fact or behaviour.
Self-contained means the grader has enough context
Write the expected answer or evidence into the criterion where practical. “Mentions the correct date” is weak if the grader has to search elsewhere to learn what date is correct. “States that the deadline is 30 June 2026” is easier to apply, assuming that date is established by the task source.
Avoid private shorthand, unexplained acronyms and references such as “as above”. If external evidence is needed, identify the authoritative source and the relevant passage.
A repeatable drafting method
First, read the original task and authoritative reference material. List the facts or behaviours an ideal answer needs. Draft one criterion for each distinct item. For each, specify an observable pass condition, then test it against one answer that should pass and one that should fail.
Finally, ask another reviewer to score the same examples without your help. If their decisions differ, tighten the criterion or split it. Check that the full set covers the task without double-counting.
What to avoid
Do not reward confident wording when the fact is wrong. Do not make a criterion depend on information unavailable to the person being graded. Avoid criteria that penalise harmless wording differences when the meaning is equivalent. Keep mandatory safety or eligibility checks separate from style preferences.
Questions to ask before you start
- Does this criterion test just one observable thing?
- Could a reviewer judge it without reading the rubric writer’s mind?
- Does it say what evidence counts as meeting it?
- Is it grounded in the actual task and reference material?
- Could two criteria award credit for the same fact?
Further reading: OpenAI HealthBench paper on self-contained, objective rubric criteria ↗