Scorecards for agentic behavior

When an agentic application gets something right, there are two very different things that could have happened. It might have reasoned about the request, chosen the correct tool, passed it the right arguments, read the result, and produced an answer grounded in what it retrieved. Or it might have called the wrong tool, ignored the error, guessed from memory, and landed on a correct answer by luck. From the final output alone, the two are indistinguishable. A scorecard for agentic behavior exists to make them distinguishable — to report not just whether the answer was right, but whether the behavior that produced it holds up when you look at the steps.

That shift — from grading the output to grading the path that reached it — is what separates an agent scorecard from an ordinary test report. This article is the concrete how of it: what parts of a trajectory are actually checkable, what the resulting scorecard shows, and, just as important, what a green one does not license. It is a companion to no self-approval, which covers who is allowed to turn evidence into a gate; here the subject is the evidence itself — the readable, bounded claim EvalGlass hands back after it inspects an agent's run.

Why the final answer is not enough

Final-answer correctness is a thin signal for anything that takes more than one step. A single-turn classifier can be judged on its output and little is lost. An agent that plans, calls tools, retrieves documents, recovers from failures, and threads state across a session cannot, because the output is the last link of a chain and the chain is where the risk lives. An agent that reaches the right answer through a broken process has not earned your trust; it has deferred the failure to a case the demo never exercised, fragile in a way the final answer will not reveal until production does.

This is also why a scorecard is the right artifact rather than a single pass/fail. Behavior at this granularity is not one fact; it is several, and they can disagree. Tool selection can be perfect while retrieval grounding regresses; a session can succeed end to end while one step inside it quietly errored and was masked by a later recovery. A scorecard holds those facts side by side, each with its own value, status, and honest record of what could not be measured — the point is not to compress agent quality to a number but to make it reviewable without overclaiming.

A scorecard for agentic behavior is a bounded claim about a path, not a stamp on an outcome. It says what the run measured and what it could not — never that the agent is correct or safe.

The labs building these systems make the same case. Anthropic's engineering team describes a transcript as “the complete record of a trial, including outputs, tool calls, reasoning, intermediate results, and any other interactions” — the trajectory, not just the answer.¹ They also warn against grading too rigidly: checking that an agent followed one exact sequence of calls is “too rigid and results in overly brittle tests.”¹ A good agent scorecard threads that needle — it measures whether the path was sound without demanding it be identical every time.

The checkable surface of a trajectory

“Evaluate the agent's behavior” is too vague to act on. What you can actually evaluate is a handful of concrete, observable properties of a run, each answerable from the trace the agent left behind. Naming them precisely is what turns a fuzzy worry into a metric that can score.

Each row is a different slice of the same run, and a scorecard can carry several at once. Tool selection and argument correctness are properties of individual calls; step order, retrieval use, and recovery are properties of how calls relate to one another; the session outcome is the whole arc. None of them is the final answer, and that is the point — they are the structure underneath it. A regression in any one of them is a real signal even when the final answer still looks right, which is exactly the failure mode the output alone hides.

Observable, not self-narrated

Here is the line that does the most work, and it is EvalGlass's own design stance. Every property above is scored from observable artifacts — the exported trace, the actual tool calls and their arguments, the retrieved documents, the real state the run produced — and not from the agent's own account of what it did. An agent that narrates “I searched the docs and confirmed the answer” is producing a claim, not evidence. The scorecard treats that narration as untrusted input and grades the call that actually fired, the arguments it actually carried, and the result it actually returned.

The sibling article on no self-approval states the rule directly: wherever it can, EvalGlass “scores observable artifacts — exported traces, tool calls, actual state — over an agent's self-narrated account of its own progress, because what an agent says it did is itself evidence, not authority.” For an agent scorecard this is not a side note; it is the foundation. A check that read the agent's claimed progress would be measuring how persuasive the agent is, not what it did — and a capable model can be very persuasive about a step it skipped.

Two boundaries follow from this and are worth stating plainly, because the temptation to overclaim runs both ways. First, EvalGlass does not inspect a model's private or hidden reasoning, and it does no chain-of-thought monitoring; the subject is the trace of what the agent did, not a window into what it was thinking. Second, those traces are imported from exports today — OpenTelemetry, OpenInference, or local JSON, with no SDK and no network on the run path; live pulls from tracing platforms are an experimental, opt-in lane. The observable surface is exactly the recorded behavior you already have, mapped to one evaluable shape on your own machine.

1. plan

   Form an approach

   Check: tool selection — was the right tool chosen for this request, read from the call that fired.

2. tool call

   Invoke a tool

   Check: tool arguments & step order — the actual arguments and sequence, never the agent's description of them.

3. retrieval

   Read what came back

   Check: evidence use — was the answer grounded in the documents actually retrieved?

4. recovery

   Handle a failure

   Check: recovery / fallback — on an error or empty result, did it notice and recover, or push on regardless?

5. outcome

   End the session

   Check: session outcome — did the whole arc reach the goal, end to end?

Reading a scorecard, field by field

The artifact EvalGlass hands back is scorecard.json — the typed source every report, terminal summary, and CI annotation renders from. Read it in one direction: verdict first, then why. The verdict tells you what CI should do; the per-metric status counts tell you what was actually measured; the authority block tells you whether any gate was even active. The trap to avoid is reading informational as pass, or reading a missing measurement as a low score. The structure is built so those two misreadings are hard to make.

A single Score inside the card carries four fields, and reading all four is what keeps a number from lying. A value — the measurement itself, when there is one. A status — one of scored, blocked, non_evaluable, skipped, or error. A note on validity — whether the measurement can be trusted and against what it was taken. And diagnostics — the why behind the verdict, the part you read after the verdict itself. The rule that matters most here: a metric that did not score carries value=null, never a fabricated 0.0. A recovery check on a session that never errored is reported as non_evaluable with a null value — the absence of a measurement, not a measurement of bad recovery. The exact shape of a Score and how it resolves is in the runtime reference.

From that same typed source you can read the card two ways. The per-metric view holds one property — say, evidence use — across every unit it applies to, so a single behavior's distribution sits in one place. The per-call view (/evalglass view --by-call) pivots the other way, grouping results by explicit subject identity (example_id / unit_id) so you can line a specific call up against its baseline — grouping by identity, never by list order, and showing a call whose metric did not score with that status rather than as 0.0. Each agent check from the table above lands as one metric in this structure, so “tool selection stable, retrieval grounding regressed” reads off the card directly.

One more honesty property matters most when the subject is agent behavior: a delta against a baseline is only a regression when the two runs are comparable. If you changed the model and the dataset at once, EvalGlass says the runs are not comparable and names the dimension that differs rather than calling the change a regression. Together with the null-not-zero rule, this keeps the gaps in agent coverage visible instead of papering over them with a confident-looking number.

None of this removes you from the loop. A scorecard is bounded evidence for a decision a human owns, not a replacement for judgment — OpenAI's evaluation guidance makes the same point, that you should “combine metrics with human judgment to ensure you're answering the right questions.”² The card sharpens the question and reports what it could measure; deciding whether those measurements are the right ones, and whether any of them earns a gate, stays with you.

Naming the grain a score speaks for

The last thing to read on an agent scorecard is the grain — the slice of behavior a score actually speaks for. EvalGlass names it explicitly as one of call, step, trajectory, or session, so that a call-level score is never read as a statement about the whole agent. As the reference puts it, “0.95 on 200 call units” and “0.95 on 12 session units” are very different claims even though the number is the same. A trajectory or session unit lists the units it spans in its members and shares a trace_id, so an aggregate metric like trajectory_shape can report, say, the fraction of members that produced a non-null output — a genuinely path-level measurement, not an output-level one.

It is worth being honest about the ceiling here, since the grain is where overclaim is easiest. The common, always-available unit is call; richer units are reported only when a run actually selects them, never implied. And when you ask your agent to group results more finely, /evalglass view --by-call groups scores by their explicit subject identity (example_id / unit_id) — grouping by identity, never by list order, and showing a call whose metric did not score with that status rather than as 0.0. Mapping a score all the way back to the source function that produced the call is not built: it would need a trace-to-call-site correlation that does not exist yet, and inventing one would be the false confidence the project forbids. Per-call grouping by explicit identity is the honest ceiling today.

Four verdicts, and what renders versus what decides

Roll the per-metric scores up and a run resolves to exactly one of four verdicts. informational — reported, but nothing gates on it; this is what a fresh run is, by design, because authority starts empty. pass — every gating metric met a threshold you approved. fail — a gating metric was validly measured and missed it. blocked — a gate cannot be claimed honestly (evidence, comparability, calibration, or a required measurement was missing), so EvalGlass holds the gate open rather than guessing. The precedence is fixed: blocked outranks fail, which outranks pass, which outranks informational. In CI, informational and pass both exit 0, fail and blocked exit 1, and an infrastructure error exits 2. The thing to hold onto is that exit 0 is not the same as “quality checked”: an informational run exits cleanly precisely because it decided nothing, and there is no gate, approve, or certify verb anywhere for an agent to call.

It also matters which thing in the system does the deciding, because three artifacts are easy to confuse. The scorecard.json is the typed source of truth. The RunRecord is the fuller record of the run behind it. The rendered report — the terminal summary, the CI annotation, the page a person reads — is a presentation of that data: it renders, it does not decide. A single Verdict Engine is the only place in the system where a verdict is ever computed; the report displays the verdict the engine already resolved and cannot invent a different one. Change the underlying records and the report follows; the report never leads. That separation is what stops a nice-looking summary from quietly becoming the source of authority.

A summary you can show a stakeholder

Because the scorecard is a typed source rather than a screenshot, it renders into a summary you can put in front of a product lead or a client without overclaiming — which is the whole point of customer confidence. The honest version of that summary names the grain (“measured across 38 sessions and the 240 calls within them”), reports the outcome (“informational — no gate is enforced yet”), shows which properties held and which moved, and states plainly what was not measured. It reads as a bounded claim, because that is all it is: these checks, on these trajectories, against these thresholds, at this grain.

What it must not become is a stamp. The temptation, especially with a stakeholder in the room, is to let “all checks green” quietly mean “the agent is good.” A scorecard read that way has stopped being a measurement and started being a marketing artifact. The discipline is to let the document say exactly what it measured and no more, and to keep the authority to gate on the human side, where it belongs. A clear scorecard makes the conversation with a client easier because it is honest about its own edges, not despite it.

Sources

Field context for why agent behavior is evaluated at the trajectory level. Neither source describes EvalGlass; the observable-over-narrated stance and the bounded-claim scorecard are EvalGlass's own design, and EvalGlass does not solve the problems these sources document.

1. Anthropic — Demystifying evals for AI agents (2026): a transcript is the complete record of a trial — outputs, tool calls, reasoning, intermediate results; without evals, teams can't distinguish real regressions from noise; checking one exact tool-call sequence is too rigid and brittle.
2. OpenAI — Evaluation best practices: combine metrics with human judgment to ensure you're answering the right questions — automated scores alone are insufficient.