Evaluating agentic workflows

The most natural way to judge an AI feature is to look at what it said. You ask it a question, you read the answer, and you decide whether the answer is good. That instinct is correct for a single model call, where the input and the output are the whole of the behavior. It quietly stops being enough the moment the feature takes more than one step — and most useful AI features now do. They plan, choose and call tools, retrieve context, recover from errors, and chain several moves before they ever produce the thing you read. A score on that final thing tells you the destination was reached. It says nothing about the path taken to get there, and the path is where agents fail.

This is the difference between scoring an output and evaluating a workflow. Evaluating the workflow means naming which slice of behavior you actually measured and asking the harder questions that only the path can answer: did it reach for the right tool, pass it well-formed arguments, ground its answer in what it retrieved, follow the steps it was supposed to, recover sensibly when a call failed — and reach the outcome for the right reasons. This article is about the system under that lens, and about moving past output-only scoring without overclaiming what any one measurement establishes.

An AI application versus an agentic workflow

It is worth being precise, because the two terms get used loosely. An AI application is the whole product — the user-facing thing your customers touch, which may be as simple as a single prompt wrapped in a UI. An agentic workflow is the part of that application that takes more than one step under its own direction: it decides what to do next based on what just happened, calls tools, reads their results, and continues until it believes it is done. A summarizer that makes one model call is an AI application but not, in any interesting sense, an agentic workflow. A research assistant that searches, reads, re-searches, and then drafts is both — and only the second kind exposes a path long enough to be wrong while still landing on a right-looking answer.

Hold that lucky-shortcut case in mind, because it is the whole reason path-aware evaluation exists. Picture a research assistant that answers a user's question correctly. Graded on the final response alone, it passes. But the trace of its run shows it called the wrong tool first, ignored the context it retrieved, and only recovered because the question happened to sit in the model's training data. The next, harder question — the one it cannot shortcut — will fail, and a final-answer score gave you no warning at all. The output was right; the workflow was broken. Evaluating the workflow is what surfaces that gap before your users do.

The EvalUnit ladder: scope before score

Before you can score behavior honestly, you have to say what slice of behavior the score speaks for. EvalGlass makes that explicit with the EvalUnit — the declared slice a result is a statement about. There are four kinds, and they form a ladder from the narrowest claim to the broadest. A call is one model call, its input and its output; this is the rung a final-answer-only score lives on, and it is the shape the default path measures. A step is one move inside a larger run — a single tool choice and the arguments it passed, for instance. A trajectory is an ordered sequence of units that share a trace_id: the rung where tool choice, step order, retrieval grounding, and recovery first become visible as a connected whole. A session is end-to-end behavior across a whole conversation or task, the outcome read in its full context.

1. 01 · call

   One model call

   Its input and output. A final-answer-only score lives here — a statement about that call alone.

2. 02 · step

   One move in a run

   A single step — for example a tool choice and the arguments it passed.

3. 03 · trajectory

   An ordered sequence

   Units sharing a trace_id. Tool choice, step order, retrieval grounding, and recovery become visible.

4. 04 · session

   End to end

   A whole session across trajectories — the outcome, in context.

The reason to declare the unit is that a number means nothing until you know what it counts. A scorecard is only ever as broad as the units it measured. If every example you ran was a single call, then a green result says those calls met their thresholds — it does not say the multi-step agent behaves correctly end to end, however tempting that reading is. The richer units in EvalGlass are explicit, never implied: a run reports trajectory- or session-level results only when it actually selects and aggregates those units. And the ladder is enforced downward, too. An aggregate metric such as trajectory_shape reads the members a trajectory or session spans; ask it to run over a bare call, which has no members, and EvalGlass reports non_evaluable rather than inventing a misleading number.

“0.95 on 200 calls” and “0.95 on 12 sessions” are different claims, even with the same number. Scope before score.

Outcome checks and process checks

With the unit named, the checks themselves fall into two families, and a mature evaluation asks both. An outcome check is a question about the destination: did the workflow reach the intended result — answer quality, task completion, the final state? A process check is a question about the journey, and it is the family that output-only scoring leaves on the table entirely. The two can disagree on the very same run. In the lucky-shortcut scenario, the outcome check passes while the process checks fail — which is precisely the signal you wanted, and exactly the signal a final-answer score destroys by averaging it away.

Process checks are not a single thing; they decompose along the moves an agent makes. The first is tool choice: did it pick the right tool for the job — reach for the search tool, not the calculator? The second is tool arguments: having chosen a tool, did it pass valid, well-formed inputs scoped to the question actually asked? The third is step order and recovery: did it follow the sequence the task required, and when a call errored, did it retry or replan rather than fabricate past the gap? The fourth is retrieval grounding and policy adherence: did the answer cite what came back from retrieval instead of ignoring it, and did the whole run stay inside the constraints you set? That last pair is where retrieval-augmented systems live or die — the faithfulness of an answer to its retrieved context is a process question, never an outcome one.

Where each check lands on the path

These check families are not abstract; each one attaches to a specific point on the trajectory. Run the research-assistant scenario along its path — an ordered sequence of units sharing a trace_id — and the checks distribute across it cleanly. These are checks you author for your application's real behavior, not a packaged benchmark; EvalGlass scaffolds and measures them and reports what it found.

Read down the column and the asymmetry is plain: four of the five checks have already done their work before the final answer exists. An output-only score sees only the last row. That is why a trajectory-scoped result tells you something a call-scoped one structurally cannot — it is the only one with the earlier rows in view. The practical payoff shows up first in agentic app evaluation, where tool selection, step order, and outcome are graded together rather than collapsed into a single pass/fail.

Traces are evidence, not truth

To check the path, you need a record of the path — and that record is a trace: the captured behavior of a run, its model calls, tool calls, retrievals, and handoffs, in order. A trace is the subject your checks run on. The first thing to be clear about is where the trace comes from. EvalGlass reads traces you have already exported — local JSONL, or exported OpenTelemetry and exported OpenInference span dumps. That import is the part that works today, and it works with no provider SDK and no network call: you hand it a file, it maps the spans to a vendor-neutral shape at the edge, and the vendor objects never travel inward. A live pull from a running tracing platform is a separate, experimental, opt-in lane — not what trace import does now, and not presented as if it were.

The second thing to be clear about is subtler and more important: a trace is input that must be validated, not a verdict to be believed. It records what the workflow did, not whether what it did was right — only the checks you authored can say that. And because a trace is raw input, it can be malformed. Here EvalGlass refuses the easy, dishonest move. A span missing an id or an output does not silently become a low score; it becomes a typed diagnostic, trace_mapping_incomplete, counted and excluded from the result. The record reconstructs faithfully or it refuses — it never shrinks quietly, and it never fabricates a 0.0 that would read as “the agent failed” when the truth is “the evidence was incomplete.” That distinction — a gap reported as a gap, never as a failure — is what keeps trace-based scores trustworthy.

Where this sits in the field

The move beyond output-only scoring is not EvalGlass's invention, and it is worth being explicit about that. The field has converged on grading the trajectory, not just the answer: OpenAI describes trace grading as scoring an agent's “end-to-end log of decisions, tool calls, and reasoning steps,” rather than its final output alone.³ The vocabulary that makes those moves nameable comes from the observability standards. OpenTelemetry's in-development GenAI semantic conventions give spans for agent operations and tool execution;¹ Arize's OpenInference conventions define span kinds including LLM, TOOL, CHAIN, RETRIEVER, and AGENT.² Those specs are the shared grammar of an agent run; EvalGlass reads exports written in them.

What is genuinely EvalGlass's own is the posture around that grammar, not the grammar itself. The exported-only, no-SDK, no-network stance is a deliberate design choice, not anything the specs require. So is the refusal to fabricate a score from a malformed span, and so is the rule that the tool measures and reports while a human, through reviewed records in the repo, decides and gates. EvalGlass is pre-alpha: Claude Code is the primary runtime, Codex is supported with a public listing still to come, exported-trace import works today while live tracing-platform pulls are planned, and there is no hosted platform and no telemetry. The category — path-aware evaluation of agentic behavior — is real, practiced, and grounded in the sources below; the specific way EvalGlass keeps it honest is the part to judge on its own terms. For the deeper claim underneath the gate, see no self-approval: why trace evidence still is not authority.

Sources

The vocabulary of agent execution and the principle of grading the trajectory come from observability standards and lab guidance. EvalGlass's exported-only, no-SDK posture — and its refusal to fabricate scores from malformed evidence — are its own deliberate choices, not requirements of these specs. None of these sources describe EvalGlass.

1. OpenTelemetry — GenAI semantic conventions (in development): names spans for agent operations and tool execution.
2. OpenInference semantic conventions (Arize): span kinds including LLM, TOOL, CHAIN, RETRIEVER, and AGENT.
3. OpenAI — Trace grading & Evaluate agent workflows: grade an agent's end-to-end log of decisions, tool calls, and reasoning steps.