Build checks

Calibration

When judge evidence can gate.

An LLM-as-judge metric may gate only with host calibration plus an approved threshold. An uncalibrated judge stays informational; a drifted or retired judge blocks an active gate. Judge evidence never authorizes itself — and calibration authorizes gating against your thresholds, it never certifies the AI.

A judge model is useful, but it is also opinionated, non-deterministic, and easy to over-trust. So EvalGlass treats a judge’s output as evidence, not authority: the Runtime Harness collects it, the core parses it into a Score, and Authority only lets it gate once a host has calibrated it against known data and approved a threshold.

The calibration states

State	Meaning	Effect on gating
uncalibrated	No calibration record exists yet.	Keeps the metric informational.
calibrating	Calibration is in progress.	Still informational.
calibrated	A host calibration record is in force.	May gate — if the threshold is also approved.
drifted	The judge no longer matches its calibration.	Blocks an active gate.
retired	The calibration was withdrawn.	Blocks an active gate.

Note the asymmetry: an uncalibrated judge falls back to harmless evidence, but a judge that was trusted and then drifted is treated as a hazard — it blocks rather than quietly keeps gating on a calibration that no longer holds.

What a calibration record must carry

Calibration is a deliberate, host-owned act. A record reaches calibrated only with an approver, a written rationale, and observed variance across at least two runs — enough to show the judge agrees with known answers and is stable. A evalglass.yaml can never declare itself calibrated; an unbacked judge metric is forced to uncalibrated.

Calibration is a prerequisite, not a formality

A judge can be scored and produce a perfectly valid number, and still cannot gate while it is uncalibrated. Even if evalglass.yaml asks the judge to gate, the run stays informational until the host supplies calibration. The configuration can request a gate; it cannot conjure the calibration that authorizes one.

yaml

# the config asks the judge to gate…
metrics:
  helpfulness:
    judge: relevance-v1
    metric_status: gating
    threshold_approval: approved
# …but the judge record is still uncalibrated, so the run stays informational —
# the judge is scored, the gate never activates.

The plugin surfaces this as two distinct, non-authoritative steps: /evalglass add-judge scaffolds an uncalibrated judge (evidence, never a gate), and /evalglass calibrate records host-owned calibration evidence. Neither self-approves: calibrate writes what was observed, and only a host-signed calibration record moves the judge to calibrated.

A failed judge is not a low score When a judge call times out, errors, or returns malformed evidence, that becomes typed infrastructure evidence — the metric is blocked or error, never 0.0. A judge that could not answer is not the same as a bad answer.

Next steps

Build checks

Judge metrics

Evidence first, calibration second, gating last.

Trust Model

Authority

Where calibration feeds the gating decision.

Runtime Harness

Judges

How judge evidence is collected.

JudgeCalibration lives in src/evalglass/core/authority.py; calibration records are host-owned under evals/calibration/ (CLAUDE.md §14).