Runtime & Integrations · Evaluation Core
Metric types and MetricSpec
Choose, declare, and implement a measurement.
MetricSpec, validated by the MetricRegistry,
and produced by an evaluator — a built-in or a host function. An evaluator
receives data and returns a Score; it never returns a verdict, and it never returns
a false 0.0.
Adding a metric is three decisions: which lens
it uses, what its MetricSpec declares, and what evaluator produces it. The registry
checks that declared score names, types, ranges, directions, aggregation, evaluator refs, and
authority requirements are all consistent before a run starts.
The MetricSpec contract
A MetricSpec declares a metric’s full meaning:
- name / version, lens, granularity, score type / range / direction;
- evaluator ref, profile, required evidence, prerequisites;
- aggregation, threshold, authority, and data policy.
The registry rejects a spec whose emitted score names, types, or aggregation are inconsistent — so a metric cannot quietly produce a value it never declared.
The built-in evaluators
| Evaluator | Lens | Produces |
|---|---|---|
exact-match | reference | Binary 1.0 iff output equals reference; no output/reference ⇒ non_evaluable. |
set-overlap | reference | Jaccard of whitespace token sets, 0–1; two empty sets ⇒ 1.0; no reference ⇒ non_evaluable. |
field-presence | non-reference | Fraction of host required_fields present in a mapping output; bad config / non-mapping ⇒ non_evaluable. |
structural-shape | non-reference | Binary 1.0 iff output is a mapping; absent output ⇒ non_evaluable. |
judge-score | — | Parses EvidenceBundle.judge_evidence; required-but-missing ⇒ blocked, unparseable ⇒ error. |
trajectory-shape | non-reference | Aggregate over unit.members: fraction with non-null output; call-unit/no-members ⇒ non_evaluable. |
These are the canonical registry ids exact_match@1, set_overlap@1,
field_presence@1, structural_shape@1, judge_score@1, and
trajectory_shape@1 — the same six the metrics guide
enumerates. answer_nonempty is a host evaluator example, never a built-in.
Writing a host evaluator
python
Evaluator = Callable[[Example, EvaluatorContext, EvidenceBundle], Score | ScoreBatch]
A host evaluator gets the Example, a context, and the EvidenceBundle —
data, not adapters. It returns a measurement. When evidence is missing it returns
non_evaluable, never 0.0. Host evaluators are
referenced as path/to/file.py:function and loaded by the Runtime Harness.
_evalglass.core and stays in host-owned evals/evaluators/.