Runtime & Integrations · Runtime Harness · M4

Judges

How an LLM-as-judge metric can ever gate.

The Harness collects judge evidence and fingerprints it; the host calibrates it; only then may a judge metric gate. A judge that times out, errors, or returns garbled output produces typed evidence — never a low score.

Judge metrics are the easiest place to over-trust a model, so authority is earned in three steps: evidence first, calibration second, gating last. The YAML can never declare a judge calibrated or a threshold approved; those are host records.

The flow

Step	What happens
Collection	`collect_judge_evidence(...)` is policy-aware: a non-egress source yields `JudgeEvidenceStatus.MISSING` and makes no call.
Calibration	Host calibration/*.json → a `CalibrationRecord` (+ optional `ApprovedThreshold`), requiring an approver, a rationale, and variance over at least two runs to reach `CALIBRATED`.
Rubrics	`load_rubric(...)` reads host rubrics/, fingerprinting content — a rubric edit enters gating provenance and breaks comparability.

The core evaluator then parses the collected JudgeEvidence into a Score; missing required evidence is blocked, unparseable is error. See Judge Calibration for the gating states.

Required runs use a fake judge The required test path uses fake judge evidence — no network, no provider keys. Live providers arrive as the optional live-judge lane, off by default.

Command judge

The command adapter is SubprocessJudgeModel (adapters/judge_subprocess.py, ADR 0042). It runs a host-owned judge — evals/judges/<name>.py — as a child process inside a config-driven run, selected under judge: as {adapter: command, command: [...argv], timeout_seconds: N}. It is an evidence source, not authority: the contract is JSON over stdin/stdout — stdin {example_id, metric, input, output, reference, rubric}, stdout {"value"|"score": 0..1, "rationale"}. It runs with shell=False (the argv is the whole trust surface), reads the rubric from the path-contained rubric_ref, and inherits the upstream egress policy of collect_judge_evidence (a forbidden source is never sent). Every failure — PROVIDER_ERROR (spawn/non-zero exit), TIMEOUT, MALFORMED (non-finite/missing score) — is non-OK JudgeResult evidence with no value. It stays uncalibrated → informational until the host computes an agreement study; it does not gate.

OpenAI-compatible lane

The openai lane is OpenAICompatibleJudgeModel (adapters/judge_openai.py, ADR 0040) — a second optional lane behind the same JudgeModel port as the host-endpoint live-judge lane. It is generic transport only: standard-library urllib (HTTPS-only) POSTing to an OpenAI-compatible /chat/completions endpoint, parsing choices[0].message.content with a NaN/inf-rejecting json.loads and markdown-fence tolerance. It imports no provider SDK, so the required tier stays hermetic and deleting the file leaves the required suite green. The per-metric rubrics are host-injected domain content (a missing rubric falls back to a neutral construct prompt, never an error); input/output travel as capped data in the user turn under a system instruction that treats them as data, not instructions. A missing prerequisite (no endpoint, non-HTTPS endpoint, no model) raises MissingPrerequisite and the lane skips. It returns JudgeEvidence only — no authority; calibration and threshold approval (ADR 0015) still govern whether the metric may gate, through the one Verdict Engine.

Next steps

Trust Model

Judge Calibration

The states that decide gating.

Guide

Add a Judge Metric

Evidence → calibration → gate.

Integrations

Optional Lanes

The live-judge lane.