Build checks

Discover metrics

Find out what to measure — for your app, not a benchmark.

Before you can measure quality, you have to know what to measure — for your app, not a generic benchmark. Two different things answer that. EvalGlass Core does the mechanical part: a read-only call-site scan with /evalglass setup. Deciding what to measure — deriving a failure-mode-anchored metric catalog from your code, output schemas, prompts and traces — is EvalGlass Discovery, a separate commercial product (private preview). In Core you author metrics explicitly, select generic starter packs, or run a catalog Discovery produced; whatever a producer proposes lands proposed / informational until you validate it — a proposal never approves itself.
Core executes; Discovery finds Core deliberately does not inspect your application to infer what should be evaluated. The application-specific metric discovery described in §2 is the EvalGlass Discovery product — Core is the execution target that runs the catalog it emits. See EvalGlass Discovery.

1 · Call-site scan (Core)

/evalglass setup (the installing-evalglass skill) runs a conservative, read-only scan of your repo — Python ast over source text, respecting .gitignore — to find candidate LLM call sites (provider-neutral hints such as chat.completions.create, messages.create, responses.create), flag trace directories, and surface undeclared data-policy boundaries. It never imports or runs your code, and it loads no LLM SDK.

A candidate inventory, not a guarantee Discovery reports what it could find by static inspection — it is not a promise that it found every LLM call in your app. You confirm the list; nothing is scaffolded behind your back.

2 · Metric discovery (EvalGlass Discovery)

A separate product — private preview Application-specific metric discovery is EvalGlass Discovery, not a Core capability. It is proprietary and pre-alpha; nothing here ships in the open-source Core plugin. The description below is what Discovery proposes — Core is the open execution target that runs it.

Finding call sites tells you where the model runs. Metric discovery tells you what to measure. EvalGlass Discovery deep-reads your repo, output schemas, prompts, domain notes, and any traces — across any agentic framework (LangChain, LangGraph, LiteLLM, LlamaIndex, DSPy, CrewAI, …), not just direct SDK calls — then proposes metrics anchored to the ways your app actually fails, emitted as a typed MetricCatalog you review, edit and export as runnable Core assets. Everything it proposes lands proposed and informational; a proposal is not a finding, and you accept every candidate.

A structural floor from your schema — no traces required. Two of those inputs are mineable automatically. From an app’s output schema (with_structured_output(Model)) discovery drafts deterministic contract-tier checks — structural_shape, field_presence, numeric_bounds, enum_membership — and from a long system prompt it proposes the rules the prose states but the code doesn’t (word limits, “only extract what’s explicitly stated”) as advisory suggestions. That is the honest floor: generic structural checks that catch a malformed output, never a claim that the answer is good — but discovery returns real, reviewable metrics even for an app with zero recorded traces.

Concretely, that floor is entirely contract-tier and runtime-deterministic: every check maps to one fact your output schema already declares, and answers only the Contract question below (is the output the right shape and type?) — never whether the answer is right.

From your output schemaDiscovery draftsDimension · tier
the output is a well-formed objectstructural_shapeContract · runtime
the required fields are presentfield_presenceContract · runtime
a ge/le-bounded number stays in rangenumeric_boundsContract · runtime
a Literal[…] field holds an allowed valueenum_membershipContract · runtime

Each discovered metric is placed on a quality dimension — the question it answers:

DimensionAsks
ContractIs the output the right shape and type?
CorrectnessIs the answer right against a known reference?
FaithfulnessIs it grounded in the context it was given?
CompletenessDid it cover what was asked?
PrecisionDid it avoid adding what wasn’t asked for?
CalibrationDoes its confidence match its correctness?
Domain-soundnessDoes it hold up against your domain’s rules?

Those are seven of an eleven-dimension taxonomy. Each metric also carries a tier, which decides what evidence it needs before it can gate:

  • Runtime-deterministic — a built-in or host check; no reference or judge needed.
  • Reference-gold — compares to validated reference data you own.
  • Judge-rubric — an LLM-as-judge metric that must be calibrated before it can gate.
Discovered is not approved Everything metric discovery proposes lands proposed / informational: no dataset validated, no threshold approved, no judge calibrated, nothing gating. It writes a reviewed-by-you starting point — see Metrics to shape a check and Promote a gate to enforce one.

Next steps

Build checks
Metrics

Shape a discovered metric into a check.
Build checks
Datasets

Give a reference metric validated data.
Ship
Promote a gate

Turn a proposed metric into a gate you approve.

Call-site discovery and schema/prompt mining live in src/evalglass/installer/ (discovery.py, schema_mining.py, prompt_mining.py); the metric catalog in src/evalglass/discovery/ (ADR 0013, 0039).