Reference
Scorecard JSON
The primary summary artifact, field by field.
scorecard.json is the typed summary every report renders from. EvalGlass ships
dataclass contracts with to_dict / from_dict validation — this is a JSON
contract, not a separate JSON Schema file.
Canonical snapshot
json
{
"verdict": {
"verdict": "informational",
"ci_should_fail": false,
"passing_gates": [], "failing_gates": [], "blocked_gates": [],
"informational_metrics": ["exact_match", "structural_shape", "field_presence", "answer_nonempty"],
"reasons": {}
},
"metrics": [
{ "metric": "exact_match", "aggregation": "mean", "value": 1.0,
"included_count": 3, "status_counts": { "scored": 3, "non_evaluable": 2 } }
],
"authority": {
"exact_match": { "can_gate": false, "blocked": false, "level": "informational",
"reasons": ["metric_status=informational", "threshold_proposed", "dataset_proposed"] }
},
"baseline_state": "comparison_not_requested",
"diagnostics": [],
"clusters": []
}Fields
| Path | Type | Meaning |
|---|---|---|
verdict.verdict | enum | Verdict: informational / pass / fail / blocked. |
verdict.ci_should_fail | bool | The only CI signal; true for fail/blocked. |
verdict.passing_gates · failing_gates · blocked_gates | string[] | Metric names per outcome. |
verdict.informational_metrics | string[] | Metrics that did not gate. |
verdict.reasons | object | Per-metric typed reason codes. |
metrics[].metric | string | The metric name. |
metrics[].aggregation | enum | Aggregation used. |
metrics[].value | number | null | Aggregated over scored + valid; null when none — never 0.0. |
metrics[].included_count | int | Examples that entered the math. |
metrics[].status_counts | object | Every status, counted (incl. excluded ones). |
authority[m] | object | can_gate, blocked, level, typed reasons. |
baseline_state | enum | BaselineState. |
diagnostics | array | Run-level structured diagnostics. |
clusters next | array | Additive (diagnostic clusters): failing/non-scored items grouped by shared Diagnostic.code, per metric. Emitted only when non-empty; recompute-verified on load. Built in the framework, not yet in the installed plugin. |
No per-call identity here
These metric objects are aggregates and carry no per-score subject identity. Per-call identity
(
example_id / unit_id, what view --by-call groups by) lives
on the per-score records in RunRecord JSON.