Runtime & Integrations · Evaluation Core

Aggregation

Reduce per-example scores — honestly.

The danger in aggregation is silent loss — folding a blocked or errored example into the average as 0.0, or hiding that it was excluded. EvalGlass does neither: excluded scores never touch the value, and they remain visible in status_counts so a summary can’t pretend the run was complete.

The rules

• Only scored + valid reduce. Everything else is excluded from the math.
• Every status is counted. status_counts records each one, including the excluded ones.
• Empty is null. A metric with no eligible scores aggregates to value=null, never 0.0.
• rate is mean-of-normalized — the mean of values normalized to their range, not a fraction-at-best.

json

{ "metric": "exact_match", "aggregation": "mean", "value": 1.0,
  "included_count": 3, "status_counts": { "scored": 3, "non_evaluable": 2 } }

Here two examples were non_evaluable (no reference) and were excluded; included_count is 3, and the count makes the exclusion auditable. The Verdict Engine uses the excluded count: an active gate with excluded examples blocks rather than passes on partial data.

Aggregating a trajectory

A traces route can declare unit: trajectory | session | step to grade a whole agent run (next; built in the framework, not yet in the version the plugin installs). This groups a trace’s calls into one aggregate unit scored by trajectory_shape@1, which reports the fraction of members that produced a non-null output. Two rules keep it honest:

• Egress is worst-of-members (fail-closed). An aggregate is egress-OK iff every member is, so a single forbidden or unknown member blocks the whole trajectory — never hidden by an average of the permitted parts.
• Empty stays honest. A trajectory where no member produced an output is non_evaluable (output_all_null), the same “never 0.0” rule the empty-metric value=null above follows.

Traces carry no validated gold, so an aggregate run resolves informational — the shape is a description of the recorded run, never a gate. The route key is documented in the unit ladder.

aggregate and AggregatedMetric live in src/evalglass/core/aggregation.py.