Evaluation discipline, without becoming an eval team.

Why this exists

The block was never belief. It was effort.

You already believe evaluation matters. What stops you is the tax it seems to charge — datasets, rubrics, labels, judge calibration, baselines, dashboards, CI — and the sense that doing it right means standing up an eval team you don’t have and don’t want. So the questions that actually matter go unanswered:

• You swapped the model to cut cost. Did quality hold — or did you just save money and lose the customer?
• The new prompt looked sharper in the demo. Did it improve the product, or only the demo?
• The workflow feels off this week. Is it real drift, or nerves? Today you find out from a user.

EvalGlass removes the effort, not the rigor. Your coding agent does the standing-up — the checks, the runs, the reading — so you stay a product team and keep the authority to decide what ships.

What you get

You get an answer — a scorecard, not a vibe.

Those questions come back answered: one scorecard with the verdict, every metric with its status, what moved since your last run, and a plain statement of what the run does not claim. It travels — paste it into a PR, hand it to a stakeholder, keep it in the repo. Here is a fuller one, the typed record behind it, and where its metrics came from.

the same run, as scorecard.json

{
  "verdict": { "verdict": "informational", "ci_should_fail": false },
  "metrics": [
    { "metric": "retrieval_faithfulness", "value": 0.91, "status": "scored",  "validity": "valid" },
    { "metric": "workflow_policy",        "value": 0.74, "status": "scored",  "validity": "valid" },
    { "metric": "citation_grounding",     "value": null, "status": "blocked", "validity": "not_measured" }
  ],
  "status_counts": { "scored": 5, "blocked": 1, "non_evaluable": 0, "skipped": 0, "error": 0 },
  "authority": { "citation_grounding": "blocked", "retrieval_faithfulness": "no_approved_threshold" },
  "baseline_state": "comparable"
}

The typed scorecard.json is the machine source of truth now; the human card above renders it, and a shareable report.html is the next canonical release. A renderer never recomputes the verdict.

Every value and field is typed — here is what each part of that record says:

• Metric & value. Each project-specific check and its score — one row per metric your agent proposed for your app.
• Status & validity. Whether a metric scored, was blocked, non_evaluable, skipped, or errored — a non-scored metric is value: null, never coerced to 0.0.
• Delta & baseline. What improved, regressed, or drifted since a run you promoted — shown only when baseline_state is comparable, so a model or prompt change isn’t called a regression by accident.
• Authority & verdict. Whether each metric can_gate, and the one typed verdict the build keys on. A fresh run is informational — measured, not approved.

What every scorecard means →

Where the metrics come from

Your agent finds what to measure — for your app.

You never wrote those checks. Your coding agent discovered them for your app — in two read-only layers, everything it proposes landing proposed / informational, never approved.

• Call-site discovery. now A conservative, read-only scan (Python ast, respecting .gitignore) finds candidate LLM call sites, flags trace directories, and surfaces undeclared data-policy boundaries. A candidate inventory, not a guarantee — it never imports or runs your code, and loads no model SDK.
• Metric discovery. next Your agent reads your code, output schemas, prompts, and any traces — across any agentic framework — then derives metrics for how your app actually fails: a typed MetricCatalog across an eleven-dimension quality taxonomy (faithfulness, completeness, domain-soundness, calibration…). With no traces yet, your output schema alone drafts a contract-tier structural floor. Built in the framework; the next canonical release.

Every discovered metric carries a tier that decides what evidence it needs before it can gate:

• Runtime-deterministic. A built-in or host check; no reference or judge needed.
• Reference-gold. Compares to validated reference data you own.
• Judge-rubric. An LLM-as-judge metric that must be calibrated before it can gate — until then it produces evidence, not authority (that is the blocked citation-grounding row above).

Discovered is not approved. A proposed metric is a reviewed-by-you starting point — informational until you shape it into a check and promote a gate.

How discovery works →

What real evaluation looks like when it’s ordinary

Four things you get — and not a claim more than each earns.

See every capability, with its status →

Use cases

Use it when your AI application changes.

Every moment, with the prompt and the scorecard →

How it works

The way your agent reviews AI behavior.

Code review catches code changes. This reviews what the AI actually does. You’ve seen what you get — here’s how it arrives: you ask, your coding agent does the work, locally, inside your repo.

This is the loop behind the scorecard above — your agent proposes the checks; you approve what gates.

See the loop in detail →

Getting started

A few things to say to your coding agent.

Install the plugin, then describe what you want in plain language — the right verb runs underneath.

See every verb & prompt →

Install in Claude Code

Two commands, then ask.

/plugin marketplace add Syntelesis-Lab/evalglass
/plugin install evalglass@syntelesis-lab

Claude Code is the primary plugin path; Codex is supported as a second coding-agent runtime running the same canonical skills (its public marketplace listing is coming). No install yet? Run the demo first: /evalglass run --example quickstart. To evaluate real behavior, export your OpenTelemetry, OpenInference, or JSON traces and hand them over — no keys, nothing phones home. All the ways in →