FIND JOBS IN AI GUIDE
How to write useful AI evaluation feedback
Useful AI evaluation feedback connects a decision to the task, the evidence and a correction. Work through a fictional example, then adapt the method to the instructions of a real evaluation task.
By Find Jobs in AI · Updated
TRY A FICTIONAL REVIEW
Which answer follows the notice?
Reference: “The collection desk is open Monday to Friday, 09:00–17:00. Bring your collection code. A photo ID is not required.”
Task: Using only the notice, say when collection closes and what the customer needs to bring.
Answer A
Collection closes at 17:00. Bring your collection code; you do not need photo ID.
Answer B
Collection closes at 18:00. Bring your collection code and photo ID.
Write a short rationale before opening the explanation. Which claims decide the comparison?
Compare your feedback with the explanation
Answer A follows the notice. It gives the stated closing time and collection-code requirement without adding an ID condition.
Feedback for B: “The notice closes the desk at 17:00, not 18:00, and explicitly says photo ID is not required. Keep the collection code, change the closing time to 17:00 and remove the ID requirement.”
Both errors are observable in the reference. A preference for more confident wording would not make B accurate. Holiday hours are not stated, so neither answer should invent them.
Start with the evidence you are allowed to use
Read the request, the supplied reference and the scoring rules before reviewing an answer. Identify the exact claim being tested. If the task limits you to a supplied document, do not fill gaps with a web search or a plausible assumption.
Keep factual accuracy, instruction following and style separate. A well-written answer can contradict the source; an awkward answer can still contain the required information. Use the project’s criteria to decide how much each issue matters.
Use the answer comparison as practice
Read both answers against the same requirements before revealing the explanation. Identify the deciding error, quote the relevant reference and write one sentence explaining what should change.
This is independent practice, not an assessment from a hiring organisation. A real project may specify a different scale, output format or rule for ties.
Use claim, evidence and correction
Claim: name the part of the response you are judging. Evidence: point to the source or observable behaviour that supports the judgement. Correction: state the change needed, without adding facts that the task does not establish.
For example: “The answer says collection is available until 18:00, but the notice gives a closing time of 17:00. Replace 18:00 with 17:00.” Another reviewer can reproduce this decision. “Bad answer” alone does not explain the problem.
If only a rating and short rationale are requested, a full rewrite may be unnecessary. If a correction is requested, preserve correct information and repair the error.
Explain uncertainty instead of inventing a rule
If a notice does not state what happens on a bank holiday, “the notice does not specify holiday hours” is a defensible observation. A confident claim that the desk will be closed would add information you do not have.
When the instructions do not settle an ambiguity, use the project’s flag or escalation route. Do not silently create a scoring rule or present an unsupported interpretation as certain.
Review your feedback before submitting
Check that each criticism refers to something the answer actually says. Make sure your quotation supports the judgement and the correction does not introduce another error. Avoid counting one mistake several times unless the rubric explicitly requires that treatment.
Use authorised practice material when describing this skill in an application. Do not include confidential assessment questions, client material or personal information. Distinguish personal practice from professional experience.
A concise feedback pattern
Claim: name the exact statement. Evidence: quote or locate the relevant reference. Correction: explain the change needed, without inventing missing information.
Questions to ask before you start
- Have I used the same criteria for both answers?
- Does the reference support the claim I am checking?
- Have I distinguished an error from information that is simply not stated?
- Is the suggested correction specific and consistent with the source?
- Have I followed the task’s rules on tools, confidentiality and response format?