Build checks
Config
Every key, its default, and its authority effect.
evalglass.yaml is host-owned and authority-safe by construction: an under-specified
metric stays informational + proposed.
The final column below says how each key moves a metric toward or away from gating. The loader
turns this YAML into typed run config — no key it omits ever grants authority by default.
Configuration is a declaration of intent, not a grant of trust. You can ask for a gate in YAML, but the run only honours it when host-owned authority records back it; otherwise the loader forces the metric back to a non-gating state. That is why a fresh config still produces an informational Scorecard.
Top level
| Key | Type | Default | Notes / authority effect |
|---|---|---|---|
run.id | string | — | Run identifier; recorded in provenance. |
datasets[].path | string | — | JSONL gold; status defaults to proposed → keeps metrics informational until validated. |
traces[].path | string | — | Recorded behavior (JSONL). |
traces[].format | enum | local | TraceFormat: local / opentelemetry / openinference. Imported from exported files — no SDK, no network. |
traces[].unit | enum | call | UnitKind: call / step / trajectory / session. A richer kind grades the whole run (next); absent is byte-identical to call. Stays informational — trace data cannot gate. |
task | object | — | Optional subprocess replay config. |
judge | object | — | Optional judge model config; cannot self-declare calibration. |
baseline | object | — | Optional baseline reference for comparison. |
output.dir | string | — | Where artifacts are written (overridable by --out). |
Per metric (metrics[])
| Key | Type | Default | Authority effect |
|---|---|---|---|
name | string | — | Declared metric name (must be unique). |
evaluator_ref | string | — | Built-in name@version or host path.py:function. |
lens | enum | — | reference / non_reference. |
score_type | enum | — | binary / continuous. |
score_range | [min,max] | — | Required for continuous metrics. |
direction | enum | higher_is_better | Which side of the threshold passes. |
params | object | — | Evaluator parameters (e.g. required_fields). |
dataset | string | — | Binds a reference metric to a dataset. |
metric_status | enum | informational | Must be gating to gate. |
threshold | number | — | The gate boundary (rejected if out of range). |
threshold_approval | enum | proposed | Must be approved (host record) to gate. |
requires_baseline | bool | false | If true, a non-comparable baseline blocks. |
rubric · calibration | string | — | For judge metrics; an uncalibrated judge stays informational. |
metric_status: gating or threshold_approval: approved is intent.
It is honored only when backed by host-owned authority records; otherwise the loader forces the
metric back to a non-gating state. See Promote a Gate.
How YAML becomes typed config
At load time the harness parses this file into typed run config, validates ranges and enums, and attaches authority-safe defaults. Generated or scaffolded data never enters as validated gold — anything the authoring tier writes lands proposed. The runtime contract behind these keys is documented in Run Config.