Build checks

Judge metrics

Evidence first, calibration second, gating last.

An LLM judge is evidence, not a verdict. So you add one in stages, and the framework refuses to let you skip ahead: /evalglass add-judge lands the metric uncalibrated, the YAML cannot declare a judge calibrated, and an uncalibrated judge cannot gate.

1 · Collect evidence

Run /evalglass add-judge to scaffold the judge metric and rubric. On the required path, a fake judge supplies deterministic evidence — no keys, no network. The Harness records each response with a JudgeEvidenceStatus; a timeout or error is typed evidence, not a 0.0.

2 · Calibrate

Run /evalglass calibrate to record a host calibration record under evals/calibration/. The verb records host-owned calibration evidence; it never self-approves. To reach calibrated the record needs an approver, a written rationale, and observed variance across at least two runs — evidence that the judge agrees with known answers and is stable. The full state machine is on Calibration.

3 · Gate

Gating is a host YAML edit, not a verb — there is no add-judge flag that grants authority. With the judge calibrated and a threshold approved, and the host having set the metric to gating (see Promote a Gate), the metric’s authority resolves to can_gate. If the judge later drifts or is retired, the gate blocks rather than trusting a calibration that no longer holds.

Judge adapters

A judge is a source of evidence. How that evidence is collected is a swappable adapter, chosen in evalglass.yaml under judge:. None of them changes the rule above — a judge stays informational until you calibrate it:

Every adapter returns evidence, never a score or authority. Swapping one in changes nothing about the gate: an uncalibrated judge cannot gate no matter how its evidence was collected.

The command judge

Your script, your model, still uncalibrated by default. The command adapter — a SubprocessJudgeModel — runs a host-owned judge you write as a subprocess inside evalglass run, so you can bring an arbitrary model, prompt, or domain preflight without a bespoke run script. The judge lives in evals/judges/<name>.py — the host-owned-truth location alongside rubrics/ and calibration/ — and you select it in evalglass.yaml under judge: (the fake adapter stays the default; you opt in):

yaml

judge:
  adapter: command
  command: [python, evals/judges/my_judge.py]  # argv, shell=False
  timeout_seconds: 120

The contract is JSON over stdin/stdout: the Harness writes {example_id, metric, input, output, reference, rubric} to the child’s stdin and reads {"value"|"score": 0..1, "rationale"} back. The command runs with shell=False, so host data can never be interpreted as a shell command — the argv is the entire trust surface. The rubric text is read from the host-owned, path-contained rubric_ref, so one script can serve many rubrics. Data-policy egress is enforced upstream in collection: a forbidden source is never sent to any judge.

Every failure edge is typed evidence, never a low score: a spawn failure or non-zero exit is a PROVIDER_ERROR, a timeout is a TIMEOUT, and malformed, non-finite, or missing output is MALFORMED — each carries no value. And like every judge, a command judge stays uncalibrated → informational until the host runs an agreement study and calibrates it. Running a real judge changes no verdict on its own — it cannot gate.

The OpenAI-compatible lane

Generic transport, no SDK. The openai lane — an OpenAICompatibleJudgeModel — is a second, opt-in live-judge lane that POSTs to any OpenAI-compatible /chat/completions endpoint (OpenAI, OpenRouter, or a local server). It complements the existing host-endpoint live-judge lane: same JudgeModel port, a different transport shape. It is generic transport only — the per-metric rubrics are domain content the host injects at construction; the framework ships no rubric and no provider SDK (standard-library urllib, HTTPS-only egress, a NaN/inf-rejecting JSON parse). It is opt-in, deletable, and authority-free: deleting the file leaves the required suite green.

The untrusted input and output travel as data in the user turn (capped), and a system instruction tells the judge to treat them as data, not instructions — the host’s output cannot redirect the judge. Credentials are handled the safe way: the endpoint and an Authorization: Bearer token come from host config by env-var name, and nothing secret is echoed into evidence. A missing prerequisite — no endpoint, a non-HTTPS endpoint, or no model — makes the lane skip cleanly rather than fail a run.

Judge collection and calibration live in src/evalglass/harness/ (CLAUDE.md §14).