Run and read
Read a Scorecard
What the result actually means.
scorecard.json is the primary artifact: verdict, ci_should_fail,
per-metric values and status counts, authority, baseline state, and diagnostics. The Markdown
and terminal views are renderings of this typed data — never the source of
authority. The friendly tour lives on the marketing
Scorecards page; this is the precise reader.
Read the Scorecard in one direction: verdict first, then why. The verdict tells you
what CI should do; authority tells you whether any gate was even active; metric status counts
tell you what was actually measured. The most common misreadings —
informational as pass, a missing
measurement as 0.0 — are exactly what these fields prevent.
The artifact
This is the quickstart’s Scorecard, trimmed to one metric for clarity:
json
{
"verdict": {
"verdict": "informational",
"ci_should_fail": false,
"passing_gates": [], "failing_gates": [], "blocked_gates": [],
"informational_metrics": ["exact_match", "structural_shape", "field_presence", "answer_nonempty"],
"reasons": {}
},
"metrics": [
{ "metric": "exact_match", "value": 1.0, "included_count": 3,
"aggregation": "mean", "status_counts": { "scored": 3, "non_evaluable": 2 } }
],
"authority": {
"exact_match": { "can_gate": false, "blocked": false, "level": "informational",
"reasons": ["metric_status=informational", "threshold_proposed", "dataset_proposed"] }
},
"baseline_state": "comparison_not_requested",
"diagnostics": []
}Field by field
| Field | How to read it | What it cannot prove |
|---|---|---|
verdict.verdict | The run outcome: informational / pass / fail / blocked. | informational is not pass — it means no gate ran. |
verdict.ci_should_fail | The single CI signal. true only for fail/blocked. | false does not mean “quality checked” — an informational run is also false. |
metrics[].value | The aggregated value over scored + valid examples. | A high value on a informational metric still gates nothing. |
metrics[].included_count | How many examples entered the math. | If it is below your dataset size, some examples were excluded — read status_counts. |
metrics[].status_counts | Every status, counted — including non_evaluable, blocked, error. | Excluded states are never folded into the value as 0.0. |
authority[m] | Whether metric m may gate, with typed reasons. | The report renders this; it can never grant authority the records don’t. |
baseline_state | Whether a regression claim is supported this run. | comparison_not_requested / not_comparable means no delta is claimed. |
diagnostics | Structured reasons for non-perfect states. | A warning here does not by itself fail CI. |
Same data, three views
The JSON above is authoritative. report.md and the terminal summary render it; with
--format ci the same verdict becomes GitHub annotations. None of them recompute the
verdict — they read verdict.verdict and ci_should_fail as-is.
- typed source
- renderings
report.md, the terminal, and CI annotations all
read scorecard.json as-is — they render the verdict, they never recompute it, so none of them
can grant authority the typed data doesn’t.value: 1.0 and still be informational.
The value says “measured well”; authority says
“allowed to gate.” Both must hold before a number enforces anything.
Coming to the scorecard
Two additions are built in the framework but not yet in the version the plugin installs — badged here, and never shown executing until they ship. Status lives on the ahead-of-code register.
| Addition | Status | What it is |
|---|---|---|
HTML report (report.html) | next | A fourth rendering of the same typed source: the verdict, per-metric values with their intervals, a “what this run does not claim” panel, and the delta from your previous run. Still a rendering — scorecard.json stays authoritative. |
| Confidence intervals & lower-bound gating | experimental | A score can carry a confidence interval and an effective sample size, and a gate can decide on the lower bound — so a perfect score over too few examples does not gate. Being hardened; not core-reliable. |
Diagnostic clusters
A metric value says that something is wrong, not what. The additive
Scorecard.clusters field groups a run’s failing and non-scored items
by their shared Diagnostic.code,
per metric — so “faithfulness = 0.82” becomes “the 18% that failed
are all missing-citation cases.” It is a lens on the same typed source: clusters
summarize failures, they never gate and never change a score. Built in the framework;
not yet in the version the plugin installs.
| What it guarantees | Why it holds |
|---|---|
| Grouping is by explicit cause, not list order | The pure cluster() function is order-invariant — any input order yields an equal, canonically-sorted list. |
| You cannot inject a cluster the data does not support | A fabricated or hand-edited cluster fails _verify_consistency, which recomputes the clusters from the persisted scores and rejects a mismatch. |
| A non-scored item is grouped, not zeroed | A blocked / non_evaluable / error item is counted by its cause and never coerced to 0.0. |
| The same typed source, one more rendering | Clusters render in both report.md and report.html from the scorecard.json field — no re-grouping, no second verdict path. |
Diagnostic.code and are
next. The call-identity “metrics explorer” — mapping a score
to its source call site — is unbuilt, and the
per-source-function view below is a deliberate
non-goal, never “planned.” The
Diagnostic Clusters reference has the typed field.
From per-metric to per-call
By default /evalglass view reports per metric. With
--by-call it groups the same scores by their subject identity —
the example_id/unit_id each score now carries (framework slice F1).
Grouping is by explicit identity, never by list order; a score with no identity (an
older artifact) is reported as such, never guessed.
json
{
"metric": "set_overlap",
"value": 0.4,
"status": "scored",
"validity": "valid",
"evaluator_version": "set_overlap@1",
"example_id": "datasets/grounded.jsonl#1",
"unit_id": "datasets/grounded.jsonl#1"
}
This is a single Score from runrecord.json (the complete record; the
Scorecard above is the compact aggregate). The identity is additive provenance:
it changes no score meaning, no aggregation, and nothing in the Verdict Engine — and the engine
stamps it, so evaluators never have to. Old artifacts without the fields still parse.
Three granularities
| Granularity | Groups by | Status |
|---|---|---|
| per-metric | metric name | shipped — view |
| per-call | explicit subject identity (example_id/unit_id) | shipped — view --by-call |
| per-source-function | the discovered source call site | non-goal |