Comparison scope

Choose a baseline and candidate

Describe the current and proposed paths precisely enough that another reviewer can tell what changed.

Audience: Engineering and AI practitioners who own model, prompt, gateway, or provider configuration.
Question: What exactly belongs on each side of the comparison?

Make the baseline real

Use the path the product relies on today, not a generic model-family name. Include the provider, account or gateway path, endpoint, model identifier, prompt version, important parameters, and output contract.

01
Record the exact provider and API path.
02
Record the model identifier visible to the approved account.
03
Name the prompt, tool, retrieval, and response-contract versions that affect the result.
04
Capture relevant retry, timeout, and fallback behaviour.

Change one decision at a time

The candidate may change a model, provider, endpoint, prompt, or configuration. If several things change together, state that the assessment evaluates the bundle and cannot attribute every difference to one part.

Separate intended differences from fairness controls

Hold the cases, quality criteria, output requirements, repetition policy, and measurement definitions steady. Document any provider-specific setting that cannot be made equivalent.

01
Mark each field as held constant, intentionally changed, or not comparable.
02
Review token limits and sampling settings for accidental advantages.
03
Confirm that both options receive equivalent context and instructions.

Worked example

Example: candidate is more than a model name

Baseline: Provider A, Atlas 2, support prompt v14, temperature 0.2, 900-token limit. Candidate: Provider B, Meridian 3, the same prompt and limits. The provider path is an intended difference; the cases and checks are controls.