What a green AI eval actually means

You changed a prompt, switched a model, or shipped a new tool, asked your coding agent to evaluate it, and the run came back green — a passing scorecard, or exit 0 in CI. Now a colleague asks the only question that matters: so, is it good? What you are allowed to say next is a smaller, more precise thing than the green light suggests, and the gap between the two is where most teams quietly overclaim. The dangerous move is to read green as “the output is correct, safe, unbiased, ready to ship.” A pass earns none of those. This article gives you a reusable reading of what a pass actually licenses, and the checks that must hold before green even means “an approved threshold was met” rather than “nothing was actually gated.”

EvalGlass is coding-agent-operated AI quality control, and its creed is no false green: a pass is a bounded claim about what was measured, never proof of correctness or safety. That creed only earns its keep if it is operational — if a green result is shaped, in code, so it cannot say more than the run established. The rest of this piece makes the boundary concrete: what a pass asserts, the four filters every claim must survive, why exit 0 is not verification, and the typed fields you read to confirm the claim before you repeat it to anyone.

The question a green run forces

Treat a green eval the way you would treat a green unit-test suite, and you will overclaim by default — because the two are not the same kind of signal. A unit test pins one input to one expected output; passing it means that exact assertion held. An AI eval measures a sample of behavior with metrics that are themselves imperfect instruments, against thresholds someone chose, on a set of examples someone supplied. Each of those qualifiers is a place the claim can leak. A green eval is therefore not a verdict on your product; it is a verdict on a measurement — and the first discipline is to keep those two apart in your own head before you put the result in front of a PM or a customer.

The honest reading is narrow on purpose. A pass means: the specific metrics that ran produced valid measurements, on the examples you gave, and those measurements met the thresholds you approved. Full stop. It does not reach the output's correctness, its safety, its fairness, or its readiness for production, and it says nothing at all about behavior the run never touched. Everything in this article is an unpacking of that one sentence, and of the structure EvalGlass uses to keep a green light from saying any more than it.

What a pass says, and what it stays silent about

The cleanest way to hold the boundary is as a two-column contrast: the left column is everything a pass earns you, the right column is everything it stays silent about. Read a green result against both at once, and the temptation to round “all checks green” up to “the AI is good” loses its grip.

The last row is the one teams miss most. Green and gating are different states. A scorecard can be entirely green and still have enforced nothing — because EvalGlass starts with empty authority, so a fresh install gates on nothing and reports rather than decides. That is the difference between “an approved threshold was met” and “nothing was actually gated,” and you cannot tell which one you are looking at from the color alone. You read it off the card, which is what the rest of this article teaches.

The four filters a claim must survive

EvalGlass turns the narrow reading into a structure: a claim becomes part of a green verdict only if it survives four filters in order. Skip any one and the “pass” you are holding is informational, not enforced — green that decided nothing. Run them in your head, in sequence, every time you read a result.

1. 01 · measurement

   Scored and valid

   The metric actually produced a value and that value is trustworthy. A non-scored metric carries a typed state — blocked, non_evaluable, skipped, error — never a fabricated 0.0.

2. 02 · authority

   Host-owned

   Validated data, an approved threshold, a calibrated judge, and an allowed data policy. Until a host act grants it, a metric is informational — it reports, it does not gate.

3. 03 · comparability

   Fingerprints match

   A delta is a regression only when the baseline fingerprints match on the gating dimensions. Otherwise EvalGlass reports not_comparable and names the dimension that differs.

4. 04 · verdict

   One Verdict Engine

   A single engine resolves exactly one outcome — informational, blocked, pass, or fail. The report renders it; it never recomputes it.

Walk the filters once. Measurement asks whether there is an honest number at all: a metric that could not run is reported with its typed status and a null value, because a missing measurement is the absence of evidence, not evidence of bad quality, and writing it as 0.0 would be a fabricated fact. Authority asks whether anyone with standing said this number may decide anything — validated data, an approved threshold, a calibrated judge, an allowed data policy, all host-owned. An uncalibrated LLM-judge metric can report but cannot fail CI; it stays advisory until you calibrate it against your own labeled examples. Comparability asks whether a delta even earns the word “regression”: if you changed the model and the dataset at once, the runs are not_comparable and EvalGlass says so rather than inventing a trend. Verdict asks whether one engine resolved the outcome — there is exactly one Verdict Engine, and the rendered report displays the verdict it already minted rather than computing its own.

The common mistake is reading green as a property of the output. It is a property of a measurement that survived all four filters — and a measurement that skipped any one of them is green that gated nothing.

Tracked end to end, that is what the opening sentence compresses: a green eval means “an approved threshold was met” only when there was a valid measurement, host-owned authority, established comparability where a delta was claimed, and a single-engine verdict — each bound, finally, to the named scope the claim speaks for. Miss any one of the four filters and you are looking at green that means “nothing was actually gated” — still useful as evidence, but not the trust signal a casual reading assumes.

Exit 0 is not verification

In CI the green light is an exit code, and an exit code carries even less than a scorecard. EvalGlass maps four verdicts onto exit codes with a fixed precedence — blocked outranks fail, which outranks pass, which outranks informational. In the shell, informational and pass both exit 0; fail and blocked exit 1; an infrastructure error exits 2. The consequence is the one to internalize: exit 0 covers both pass and informational. A fresh run, with empty authority, gates nothing and exits 0 as informational — it decided nothing and exited cleanly precisely because it decided nothing. So a green pipeline does not, on its own, mean “quality was checked.” It means “nothing failed,” which is a different and weaker claim until a gate you approved was actually active.

This is deliberate, and it is the whole point of empty authority. A tool that exited 1 out of the box — that gated on data no human had validated, against thresholds no human had approved — would be manufacturing confidence from nothing, which is the exact failure no false green exists to refuse. There is no gate, approve, or certify verb anywhere for the agent or the run to call; turning a check into a gate is a host edit you make in your own config. So the honest reading of a green CI run is two-step: first confirm a gate was active at all, then confirm the metric behind it survived the four filters. A green that skipped step one is informational wearing the color of a pass.

Reading the card to confirm the claim

Every caveat above maps to a typed field on the verdict and the scorecard EvalGlass hands back, so you read the data, not the prose. Read it in one direction — verdict first, then why — and run a short checklist before you repeat a green result to anyone.

Two notes on the grain, because it is where overclaim is easiest. You can pivot the card by explicit subject identity with /evalglass view --by-call, which groups scores by example_id / unit_id — grouping by identity, never by list order, and showing a call whose metric did not score with that status rather than as 0.0. But mapping a score all the way back to the source function that produced the call is not built; per-call grouping by explicit identity is the honest ceiling today, and inventing a finer view would be exactly the false confidence the project forbids. The scorecard is bounded evidence for a decision you own — not the decision itself, which is the same point OpenAI's evaluation guidance makes when it says to combine automated metrics with human judgment rather than lean on the scores alone.²

Where this sits against the field

The reason a green number can over-claim is not an EvalGlass quirk; it is a documented property of AI evaluation, and the boundary above is a response to it, not a solution for it. An interdisciplinary review of roughly a hundred studies found that many AI benchmarks suffer weak construct validity — they fail to measure what they claim — so a high score does not establish the capability it appears to assert.³ That is precisely why a pass must stay a claim about a measurement rather than a claim about a capability: EvalGlass does not repair the validity of your metrics, and presenting a green as proof of the thing the metric was supposed to capture would inherit exactly the gap that review describes.

The same caution sits in the standards. In the NIST AI Risk Management Framework, “valid and reliable” is the base trustworthiness characteristic, and it is assessed by ongoing testing and monitoring, not by a single passing test.⁴ A green eval is one measurement at one moment; reading it as a standing trustworthiness claim is the category error the framework is built to prevent. And the labs building these systems make the operating-level version of the same point: Anthropic's engineering team treats a transcript as the complete record of a trial and warns that automated scores alone do not let teams distinguish real regressions from noise, while grading one exact tool-call sequence is too rigid and brittle.¹ A green that only checked the final answer under-claims the behavior actually exercised — which is why EvalGlass scores the path and reports what it could not measure, rather than compressing it all to a single confident light.

None of those sources describe EvalGlass, and EvalGlass solves none of the problems they document. They are the reason the claim boundary exists: a green number is bounded evidence for a human decision, motivated by a field where scores routinely promise more than they hold. The structural separation of evidence from authority — that the actor which creates evidence cannot authorize it to gate — is EvalGlass's own design choice, covered in no self-approval, and the honest thing to evaluate it on.

Sources

Field context for why a score can over-claim and why a pass must stay bounded. None of these sources describe EvalGlass; the claim boundary, the four filters, and host-owned authority are EvalGlass's own answer to the problems they document, and EvalGlass does not solve construct validity, certify trustworthiness, or remove the human from the decision.

1. Anthropic — Demystifying evals for AI agents (2026): a transcript is the complete record of a trial; without evals teams cannot distinguish real regressions from noise, and checking one exact tool-call sequence is too rigid and brittle.
2. OpenAI — Evaluation best practices: combine automated metrics with human judgment — automated scores alone are insufficient to answer the right questions.
3. Eriksson et al. — Can We Trust AI Benchmarks? An Interdisciplinary Review (2025): many AI benchmarks suffer weak construct validity and fail to measure what they claim, so a high score does not establish the asserted capability.
4. NIST AI 100-1 — AI Risk Management Framework (AI RMF 1.0) (2023): “valid and reliable” is the base trustworthiness characteristic, assessed through ongoing testing and monitoring rather than a single passing test.