Trust Model
Baseline Comparability
When two runs can be compared.
A regression is a typed claim, not a subtraction. EvalGlass will only say one
run regressed against a baseline when their provenance fingerprints
match on the gating dimensions. Otherwise the comparison is
not_comparable,
missing_baseline, or comparison_not_requested — and no delta is claimed.
“The score dropped” means nothing if the dataset, metric, or config also changed. A regression without comparability is not a claim — it is two unrelated numbers. So comparability is resolved first, as an explicit state, and only a comparable result lets a delta carry weight.
comparison requested?
no
COMPARISON_NOT_REQUESTED
yes
baseline present?
no
MISSING_BASELINE
yes
gating dimensions match?
fail-closed: an absent dimension counts as changed
no
NOT_COMPARABLE
yes
COMPARABLE
a delta is a supported claim
- comparable
- not_comparable / missing_baseline
- comparison_not_requested
comparable blocks its authority — the gate refuses to compare across mismatched runs.The four states
| State | Meaning |
|---|---|
| comparable | Baseline present and gating dimensions match — a delta is a supported claim. |
not_comparable | Baseline present, but a gating dimension changed — the runs measure different things. |
missing_baseline | A comparison was requested but no baseline was found. |
comparison_not_requested | No comparison was asked for this run. |
Two states people merge
not_comparable | missing_baseline |
|---|---|
| A baseline exists, but it measured something different — a changed dataset, metric, evaluator, framework, config, or policy. Comparing would be misleading. | No baseline was available at all. There is nothing to compare against, so a delta cannot be formed in the first place. |
Comparability can gate
When a metric is configured to require a baseline, anything other than
comparable resolves its authority to
blocked — the gate refuses to claim a pass it cannot support,
rather than comparing across mismatched runs.