Reading a result
What a green result does not mean
No false green means a passing scorecard asserts only that measured checks cleared the thresholds you approved — never that your AI application is correct, safe, or production-ready.
A passing scorecard is a narrow, honest signal: the metrics that ran, on the examples you gave, met the thresholds you approved. It is not a verdict on your product. The value of EvalGlass is that it never lets a green run say more than the run actually proved.
A green run — what it does and doesn't license
Read a pass for exactly what it is. The left column is everything a green run earns you; the right column is everything it stays silent about.
| A green run means | A green run does not mean |
|---|---|
| The specific metrics that ran actually produced valid measurements. | Your output is correct. |
| They ran on the examples you supplied — not on everything your app does. | Your output is safe, or free of harm you didn't measure. |
| Those measurements met the thresholds you approved. | Your output is unbiased, fair, or complete. |
| Every gating metric was comparable where comparability was required. | Your app is production-ready. |
A perfect score is not a pass experimental
The sharpest proof that green never overclaims is structural, not a slogan: on the epistemic core the default gate reads the lower confidence bound of a metric’s interval, not the hopeful point — so a lucky handful of examples cannot clear it.
A perfect 3/3 does not clear the default gate
Three-for-three is a point of 1.0 — but its 95% lower bound is ≈ 0.44, so against a 0.8
threshold it does not pass. It passes only under a named point-smoke policy, a
deliberate and recorded choice. Gather more evidence and the interval tightens until the lower bound
clears the bar. Illustrative example, not a measured result.
Built in the framework; not yet in the version the plugin installs — the shipped path still gates on the point threshold. Why a perfect score can fail → · Confidence & intervals →
Non-scored states are not zero
When a metric can't honestly run, EvalGlass records why — it never invents a number to fill
the gap. A blocked, non-evaluable, skipped, or errored result is the absence of a
measurement, not a measurement of bad quality. None of these states is ever encoded as
0.0.
- blocked. A required metric could not run honestly, so it is excluded from the math and holds the gate open. Status values →
- non_evaluable. The example lacked the evidence this metric needs — missing input, not low score. An empty trajectory graded at the
unit: trajectorylevel isnon_evaluabletoo, never0.0. next Measurement state → - skipped. The metric was intentionally not run for this example or run.
- error. An evaluator or evidence parser failed unexpectedly; the failure is reported, never scored as zero.
- Per call, too. Ask your agent for
/evalglass view --by-calland it groups scores by their explicit subject identity (example_id/unit_id) — a call whose metric didn't score is shown with that status, never as0.0. The gap stays visible at every granularity. Reading a scorecard → - Grouped by cause, not zeroed. When a run’s failures are clustered (
Scorecard.clusters), each non-scored case is grouped by its cause — never coerced to0.0to pad a number. next Diagnostic clusters →
Informational is not gating
Scoring something and gating on it are two different acts. A run can measure plenty while deciding nothing, and EvalGlass keeps that line visible.
-
01 · measured
A measured valueA metric yields a value only when it scored and is valid — otherwise the result is a typed state, never
0.0. -
02 · authority
Host-owned authorityValidated data, an approved threshold, a calibrated judge, and an allowed data policy decide whether it may gate.
-
03 · outcome
One outcomeinformational— reports; never fails CI.blocked— gate held open; can't claim honestly.can_gate— threshold enforced.
informational, or blocked. How authority resolves →- A fresh run scores but gates nothing. Until you approve a threshold, every metric is informational — it reports, it never fails your build. The claim boundary →
- An uncalibrated judge can report but never fail CI. An LLM grader stays advisory until it's calibrated against your own labelled examples. Judge calibration →
- A score change is a "regression" only when the runs are comparable. If the two runs aren't comparable, EvalGlass reports that — it does not call the delta a regression. Baseline comparability →
Read your scorecard responsibly
Every caveat above maps to a typed field in the Scorecard JSON — so you read the data, not the prose. When in doubt, check the field that proves the claim.
- "What did it license?" Read the Verdict Engine output and the per-metric status — a pass binds only to gating metrics that validly met an approved threshold.
- "Is this number real?" Check the metric's status and validity before trusting its value; a non-scored state is the absence of a measurement.
- "Did anything gate?" Read the authority and threshold-approval fields — informational metrics carry no gate.
- "Is this a regression?" Read the baseline-comparability state; a delta without comparability is not a claim. Scorecard JSON reference →
The cardinal question
Could this green scorecard be misread as proof the output is correct — when the run is only informational, unvalidated, uncalibrated, non-comparable, non-evaluable, or partly blocked? If it ever could, the result is overclaiming, and that is exactly the gap EvalGlass closes.
Related