Resources/Prepare
Quality method
Define useful quality checks
Measure the behaviours the task requires instead of hiding the decision behind one generic quality score.
- Audience
- Domain experts, product owners, and AI practitioners agreeing how responses will be judged.
- Question
- What must a good response do, and what failure would matter to the decision?
Turn requirements into separate checks
A response can be correct but unusable, fluent but unsafe, or well structured but incomplete. Keep distinct requirements separate so one strength cannot erase a blocking failure.
- 01
Name required facts or actions.
- 02
Name prohibited claims, tools, or remedies.
- 03
Check the response structure when downstream code depends on it.
- 04
Add tone or style only when it changes customer or reviewer usefulness.
Choose the right reviewer for each check
Use deterministic code for exact structures and values, a qualified person for ambiguous domain judgement, and model-based scoring only where its rubric and disagreements can be reviewed.
- 01
Prefer an exact check when an exact rule exists.
- 02
Record the human role needed for policy or domain interpretation.
- 03
Test an automated judge against reviewed examples before relying on it.
Set blocking criteria before results arrive
Agree which failure prevents a change even if the candidate wins on average. This avoids moving the threshold after seeing a preferred result.
Worked example
Example: billing-dispute response
Required: explain the disputed charge and the approved next step. Prohibited: promise a credit without an approval state. The prohibited action is a blocking check; an otherwise polished response still fails.