Build checks

Metrics

Author a check — built-in or host evaluator — without breaking measurement.

/evalglass add-metric appends a MetricSpec to evalglass.yaml for you — as a proposed, informational entry. You still make two choices: pick a built-in or write a host evaluator. Whatever you write, return a Score that uses non_evaluable when evidence is missing — never a false 0.0. The verb shapes and authors a check; it never generates authoritative gold.

Not sure what to measure yet? Discover metrics proposes checks for your app first. Once you know the check you want, run /evalglass add-metric (the authoring-a-metric skill) to scaffold the entry, then choose a lens: a reference metric compares to gold, a non-reference metric inspects the output alone. The verb only proposes a metric that can report; it cannot validate a dataset or approve a threshold, so the metric stays informational until the host promotes it.

1 · Use a built-in

Six deterministic, domain-neutral, versioned built-ins ship in the registry. A missing input yields non_evaluable — never a fabricated 0.0.

Built-in	What it scores	Missing input
`exact_match@1`	1.0 iff the output equals the reference	`non_evaluable`
`set_overlap@1`	Jaccard overlap (0–1) of whitespace token sets vs. the reference; two empty sets ⇒ 1.0	non_evaluable (no reference)
`field_presence@1`	Fraction of the host’s `required_fields` present in a mapping output	non_evaluable (non-mapping / bad config)
`structural_shape@1`	1.0 iff the output is a mapping of the expected shape	non_evaluable (absent output)
`trajectory_shape@1`	Aggregate over a trajectory’s members: fraction with non-null output	non_evaluable (call-unit / no members)
`judge_score@1`	Parses judge evidence into a score; an uncalibrated judge stays informational until a host runs an agreement study	blocked if required-but-missing

These are the only built-ins — anything else is a host evaluator you write (§2 below). Full per-evaluator semantics live in the reference. /evalglass add-metric writes the reference into evalglass.yaml; the appended entry looks like:

yaml

metrics:
  - name: field_presence
    evaluator_ref: field_presence@1
    lens: non_reference
    score_type: continuous
    score_range: [0, 1]
    params:
      required_fields: ["answer"]

Every key and its authority effect is in Config.

2 · Write a host evaluator

When the measurement needs your domain knowledge, write a function in evals/evaluators/ and reference it as path.py:function. The answer_nonempty example below is a host evaluator you author — not a seventh built-in:

python

from _evalglass.core import Score, ScoreStatus, Validity

def evaluate(example, context, evidence):
    out = example.output
    if out is None:                       # no evidence to judge
        return Score(metric="answer_nonempty",
                     status=ScoreStatus.NON_EVALUABLE,
                     validity=Validity.NOT_APPLICABLE, value=None)
    ok = bool(str(out).strip())
    return Score(metric="answer_nonempty", status=ScoreStatus.SCORED,
                 validity=Validity.VALID, value=1.0 if ok else 0.0)

The evaluator receives data — the Example, a context, the EvidenceBundle — not adapters, so it cannot perform an effect. Return non_evaluable when the evidence is absent; a real 0.0 means “measured, and it scored zero,” which is a different claim.

A new metric starts proposed and informational /evalglass add-metric only proposes a metric. It lands proposed / informational and can report a value — it cannot gate. There is no verb that approves it: the metric becomes a gate only when the host validates its dataset, approves a threshold, and edits the YAML status — see Promote a Gate.

Next steps

Build checks

Judge metrics

When the check is an LLM-as-judge.

Ship

Promote a Gate

Make the metric enforce CI.

Trust Model

Measurement State

Why non_evaluable beats 0.0.