Trust Model

Drift artifact

The typed drift.json, and what licenses a regression. next

drift.json is the typed artifact evalglass watch writes each cycle: a DriftResult holding a comparability state and, when comparable, the item-paired per-metric comparison. It carries no verdict, no exit class, and no authority — evidence, never a decision. Built in the framework; not yet in the version the plugin installs.

A drift check informs; it never decides. A regression is licensed only when the runs are comparable and the paired interval clears zero — the same interval-licensing rule the Decision Policy uses, reused here, never re-implemented.

The drift.json artifact

A DriftResult serialized to reports/<run-id>/drift.json via the result store’s atomic writer. Deliberately absent: any verdict, exit, or authority field — mirroring the lane-attach seam’s “evidence, not authority” shape.

Field	Type	Meaning
`comparability`	`comparable` · `not_comparable` · `missing_baseline`	Whether the check could honestly compare the current run to its baseline (table below).
`comparison`	PairedComparison \| absent	The item-paired per-metric deltas. Present only when `comparability` is `comparable`.
`skipped_metrics`	str[] · omitted when empty	Metrics with no declared direction — compared to nothing, recorded, never a crash.

Comparability states

The current run’s typed baseline state is mapped fail-closed to one of three drift states:

State	When	Reported
`comparable`	The baseline exists and the run’s `BaselineState` is `COMPARABLE`.	A full paired `comparison` is emitted.
`not_comparable`	The run’s `BaselineState` is `NOT_COMPARABLE` (fingerprints differ).	As-is — no regression claim, never laundered into “no regression”.
`missing_baseline`	No baseline file (a legitimate first run), or comparison was not requested/possible.	As-is — a graceful state, not an error; drift simply not evaluated.

The paired comparison

When comparable, comparison is a PairedComparison — baseline_run_id plus a deltas map of one MetricDelta per metric, computed over the example ids both runs scored aggregatably.

MetricDelta field	Type	Meaning
`metric`	str	The metric this delta is for.
`n_paired`	int ≥ 0	The count of example ids both runs scored aggregatably.
`delta`	float \| null	Mean item difference (current − baseline). `null` when no items paired.
`interval`	Interval \| absent	Student-t band on the mean difference (needs `n_paired` ≥ 2); absent otherwise.
`outcome`	`regression` · `within_noise` · `improvement` · `unresolved`	The interval-licensed label (below).

What licenses each outcome

The per-metric outcome is decided by the delta’s interval and the metric’s declared direction — never by the raw sign of the delta alone:

Condition	Outcome
no interval (too few paired items, `n_paired` < 2)	`unresolved` — not enough evidence to resolve
the interval straddles zero (`lower` ≤ 0 ≤ `upper`)	`within_noise`
the interval clears zero on the better side for the direction	`improvement`
the interval clears zero on the worse side for the direction	`regression`

Drift is not a second verdict path A regression outcome is surfaced as a typed label plus an explanatory Diagnostic (INFO / WARNING) appended after the verdict — never a Verdict. The exit code still derives only from the run’s VerdictPayload; drift adds no exit class. And watch only reads the baseline for the pairing — it never promotes it. See Provenance & Baselines.

An illustrative drift.json

A comparable check with one metric flagged and one inside the noise band. The artifact has no verdict field — it is informational by construction:

json

{
  "comparability": "comparable",
  "comparison": {
    "baseline_run_id": "run-2026-06-30-abc123",
    "deltas": {
      "faithfulness": {
        "metric": "faithfulness",
        "n_paired": 24,
        "delta": -0.08,
        "outcome": "regression",
        "interval": { "method": "student_t", "level": 0.95, "lower": -0.14, "upper": -0.02 }
      },
      "helpfulness": {
        "metric": "helpfulness",
        "n_paired": 24,
        "delta": -0.01,
        "outcome": "within_noise",
        "interval": { "method": "student_t", "level": 0.95, "lower": -0.05, "upper": 0.03 }
      }
    }
  }
}

Illustrative example, not a measured result. faithfulness is a regression only because its interval clears zero on the worse side; helpfulness moved too, but its interval straddles zero, so it is within_noise. A no-comparable-regression drift never means quality is fine.

Next steps

Guide

Watch for drift

The reader’s walkthrough — run, schedule, read back.

Trust Model

Baseline comparability

When two runs can honestly be compared at all.

Evaluation Core

Provenance & Baselines

The promoted baseline the watcher reads — and only reads.

DriftResult / Comparability / evaluate_drift / persist_drift live in src/evalglass/harness/drift.py; the reused PairedComparison / MetricDelta / DeltaOutcome in src/evalglass/core/comparison.py (ADR 0048).