Trust Model
EvalUnit
What part of the agent is evaluated.
call, step, trajectory, or session. It is
how EvalGlass keeps a single-call score from being read as a statement about the whole agent.
A Scorecard is only as broad as the units it measured. If every Example is a single model
call, then a green run says those calls met their thresholds — not that the
multi-step agent behaves correctly end to end. Naming the unit keeps that scope explicit
instead of leaving it to optimistic reading.
The four kinds
| Kind | Evaluates | Read it as |
|---|---|---|
call | One model call — its input and output. | The common, always-available unit; a statement about that call only. |
step | One step inside a larger run. | Finer-grained inspection of a single move. |
trajectory | An ordered sequence of units sharing a trace_id. | Multi-step behavior; feeds aggregate metrics such as trajectory_shape. |
session | A whole session spanning trajectories. | End-to-end behavior across a conversation or task. |
call is the shape the MVP path scores by default. The richer units are explicit,
not implied: a run only reports trajectory- or session-level results when it actually selects
and aggregates those units. trajectory and session are
config-reachable — a traces route declares
unit: trajectory | session | step to group a trace’s calls into one aggregate and
grade the whole run (next; built in the framework, not yet in the
version the plugin installs). See the Trajectory & session grading
guide and the unit ladder reference.
Richer units carry their members
A trajectory or session unit lists the units it spans in
members. Aggregate evaluators read that list — for example, the built-in
trajectory_shape metric reports the fraction of members that produced a non-null
output. A call unit has no members, so an aggregate over it is reported as
non_evaluable rather than a misleading number.
call units” and “0.95
on 12 session units” are very different claims, even though the number is the same.