Build checks
Discover metrics
Find out what to measure — for your app, not a benchmark.
/evalglass setup, and
metric discovery — a catalog of failure-mode-anchored metrics your agent derives
from your code, output schemas, prompts, and traces, across any agentic framework and even before
you have a single trace — is the next canonical release. Both are
read-only, and both land everything proposed /
informational: discovery suggests what to measure, it never
approves it.
1 · Call-site discovery
/evalglass setup (the installing-evalglass skill) runs a conservative,
read-only scan of your repo — Python ast over source text, respecting
.gitignore — to find candidate LLM call sites (provider-neutral hints
such as chat.completions.create, messages.create,
responses.create), flag trace directories, and surface undeclared data-policy
boundaries. It never imports or runs your code, and it loads no LLM SDK.
2 · Metric discovery
Finding call sites tells you where the model runs. Metric discovery tells you what to
measure. The discovering-metrics skill deep-reads your repo, output schemas, prompts,
domain notes, and any traces — across any agentic framework (LangChain,
LangGraph, LiteLLM, LlamaIndex, DSPy, CrewAI, …), not just direct SDK calls — then
derives metrics anchored to the ways your app actually fails, emitted as a typed
MetricCatalog and scaffolded into runnable host assets. This is built in the framework
and is the next canonical release; today’s installed plugin does the call-site discovery above.
A structural floor from your schema — no traces required. Two of those inputs are
mineable automatically. From an app’s output schema
(with_structured_output(Model)) discovery drafts deterministic contract-tier checks
— structural_shape, field_presence, numeric_bounds,
enum_membership — and from a long system prompt it proposes the
rules the prose states but the code doesn’t (word limits, “only extract what’s
explicitly stated”) as advisory suggestions. That is the honest floor: generic structural
checks that catch a malformed output, never a claim that the answer is good — but discovery
returns real, reviewable metrics even for an app with zero recorded traces.
Concretely, that floor is entirely contract-tier and runtime-deterministic: every check maps to one fact your output schema already declares, and answers only the Contract question below (is the output the right shape and type?) — never whether the answer is right.
| From your output schema | Discovery drafts | Dimension · tier |
|---|---|---|
| the output is a well-formed object | structural_shape | Contract · runtime |
| the required fields are present | field_presence | Contract · runtime |
a ge/le-bounded number stays in range | numeric_bounds | Contract · runtime |
a Literal[…] field holds an allowed value | enum_membership | Contract · runtime |
Each discovered metric is placed on a quality dimension — the question it answers:
| Dimension | Asks |
|---|---|
| Contract | Is the output the right shape and type? |
| Correctness | Is the answer right against a known reference? |
| Faithfulness | Is it grounded in the context it was given? |
| Completeness | Did it cover what was asked? |
| Precision | Did it avoid adding what wasn’t asked for? |
| Calibration | Does its confidence match its correctness? |
| Domain-soundness | Does it hold up against your domain’s rules? |
Those are seven of an eleven-dimension taxonomy. Each metric also carries a tier, which decides what evidence it needs before it can gate:
- Runtime-deterministic — a built-in or host check; no reference or judge needed.
- Reference-gold — compares to validated reference data you own.
- Judge-rubric — an LLM-as-judge metric that must be calibrated before it can gate.