Resources/Interpret
Operational evidence
Interpret latency, reliability, failures, and cost
Keep timing, completion, recovery, usage, and price assumptions distinct so averages do not hide the trade-off.
- Audience
- Engineering, product, and business reviewers reading operational findings.
- Question
- How should the team compare speed, failure behaviour, and provider economics without overstating the sample?
Name the timing before comparing it
Time to first token, time to first usable content, and total response time answer different product questions. Report the definition, typical result, slow tail, and sample count together.
- 01
Use the same timing boundary for baseline and candidate.
- 02
Report a median and a slow-tail observation when the sample supports it.
- 03
Keep retries in the customer-experienced duration when they are part of the path.
Count every failure as an outcome
Separate rate limits, timeouts, network errors, provider errors, refusals, invalid outputs, and quality failures. Show first-attempt completion and recovered completion when retries are allowed.
Build cost from usage and a dated price basis
Use observed requests and input/output usage, then state the provider price or customer-supplied commercial assumption used. Keep the assessment-run estimate separate from a projected workload scenario.
- 01
Do not label token counts alone as cost.
- 02
Exclude unknown discounts, taxes, caching, and gateway charges explicitly.
- 03
Treat a projection as conditional on the customer’s workload mix.
Worked example
Example: faster median, slower tail
A candidate can complete the typical response faster while two rate limits make its P95 slower. The useful finding is mixed: ordinary speed improved, but recovery behaviour needs investigation.