Reference
Agent commands
One umbrella, a handful of honest verbs.
EvalGlass is a plugin you operate through your coding agent. You don’t have to
learn the verbs — describe what you want and the right skill triggers — but every act has an
explicit name, a
/evalglass <verb>, for when you’d rather drive directly. The
plugin is a typist and a reader, never a participant: it issues exact commands and reads
back typed artifacts; the host validates, and a single Verdict Engine decides.
Use natural language (“evaluate my agentic app after the model switch”) or the explicit verb — they reach the same place. What the plugin can never do is grant authority: no command approves a gate, and the absence of that verb is the identity. Install first with Get the plugin.
Try it before you install
The bundled quickstart runs with no setup — real, populated non-reference signal and an honest informational verdict, so you can see exactly how a result reads before wiring it into your own repo:
in your coding agent
/evalglass run --example quickstartStart by saying…
You don’t have to learn the verbs — describe what you want and the right skill triggers. A few starters, in the order you’ll usually reach for them; the full verb reference is below.
| You say | You get | Verb |
|---|---|---|
| “Run the EvalGlass quickstart.” | A real informational Scorecard in seconds — sample metrics, no setup, no keys. | run --example quickstart |
| “Set up EvalGlass in this repo.” | Discovered candidate call sites, the vendored runtime, and a proposed dataset, metrics, and CI. | setup |
| “Import our exported traces and build evaluation cases.” | Your exported OpenTelemetry/OpenInference or local trace JSON imported — no SDK, no network. | connect |
| “Discover metrics for my app.” | A proposed MetricCatalog of what to measure — from your code, output schemas, prompts, and traces, across any agentic framework; even your schema alone drafts a contract-tier structural floor. | discovering-metrics |
| “Evaluate my agentic app; show what improved, regressed, or drifted.” | A Scorecard with per-metric values and deltas, and an honest verdict — nothing gates until you approve it. | run → view |
| “Explain why each number is or isn’t trustworthy.” | The typed authority and diagnostics behind each metric — a missing field is reported as missing, never inferred. | explain |
| “Build checks for this workflow before we release it.” | A metric appended to your config, landing proposed / informational — it can report, not gate. | add-metric |
| “Add a faithfulness judge and calibrate it on my labels.” | A host-owned judge whose evidence stays advisory until you calibrate it. | add-judge → calibrate |
| “Compare this prompt change against the last run.” | A delta — only when the runs are typed comparable; otherwise it names the dimension that differs, never a false regression. | compare |
| “Set up a CI gate for the checks we’ve reviewed and approved.” | A CI workflow that blocks only after you approve a gate — no auto-approval, no false green. | ci |
The usual first arc is setup → connect → discover metrics,
then run. Metric discovery is the deeper, domain-meaningful layer and the
next canonical release — how discovery works →
The core verbs
| Verb | What it does |
|---|---|
/evalglass | An honest status overview — what’s integrated, your data and authority state, and the next useful step. Runs nothing. |
setup | Discover candidate LLM call sites (announced, read-only, consent-gated), vendor the runtime, and scaffold a proposed dataset, metrics, and CI. Discovery is a candidate inventory — never a promise of finding every call. |
connect | Import your exported OpenTelemetry/OpenInference or local trace JSON — no SDK, no network — or scaffold a starter dataset. |
discovering-metrics | Deep-read your code, output schemas, prompts, and traces — across any agentic framework — and draft a proposed MetricCatalog of what to measure, anchored to how your app fails. With no traces, your output schema alone drafts a contract-tier structural floor. The deeper layer after connect, and the next canonical release. Discover metrics → |
run | Run the evaluation against the vendored runtime and read back the verdict. The plugin never computes a verdict itself. |
view | Per-metric status, validity, and values from the Scorecard — never 0.0 for a metric that didn’t score. --by-call groups by each score’s explicit subject identity. The three view granularities → |
explain | Why each number is — or isn’t — trustworthy, strictly from the typed authority and diagnostics. A missing field is reported as missing, never inferred. |
compare | Show a delta against a baseline — but only when the runs are typed comparable; otherwise it names the dimension that differs, never a false regression. |
baseline | Promote a baseline — a deliberate, explicit act, never automatic. |
ci | Copy the CI workflow (an alias of setup --ci). It exits on the core’s ci_should_fail only; it adds no verdict logic and blocks only after you approve a gate. Wire CI → |
Authoring your own measurement
An opt-in tier that helps you build host-owned metrics, judges, and calibration. Everything it scaffolds lands provisional — it can report, but it cannot gate until you validate it.
| Verb | What it scaffolds |
|---|---|
add-metric | Append a metric to your config as proposed/informational — built-in or backed by your own scorer. Never an approved threshold or validated dataset. |
add-judge | Scaffold an uncalibrated LLM-judge metric and a rubric. It cannot gate until you calibrate it against your own labels. |
calibrate | Record your own calibration evidence. It never self-approves a gate — calibration is something you supply, not something the agent grants. |
Advanced lanes
| Lane | Status | What it is |
|---|---|---|
watch | next | Run one evaluation, compare it to the configured baseline, write a typed drift.json, and exit — built for cron/CI, not a resident daemon. A regression is flagged only when the runs are comparable and the paired interval clears zero; a no-comparable-regression result never means quality is fine. |
connect --live <platform> | experimental | An opt-in, deletable lane to pull from a live tracing platform, scaffolded in evalglass.yaml by the verb: credentials are env-var names (a literal secret is rejected), data_policy defaults to unknown so egress is refused until you permit it, no provider SDK is imported (a real adapter is a host extension over a stub), a missing extra is a clean SKIPPED, and a live pull is proposed and cannot gate. Removing the lane changes no verdict. |
connect --synth | non-goal | A decided non-goal, not a deferral — governance-only: there is no generator, by design. Any synthetic data is forced to proposed — it can never enter as validated gold. Why → |
view per source function | non-goal | A decided non-goal, not a deferral — grouping a score back to the source function that produced the call needs trace↔call-site correlation that does not exist; inventing one would be false confidence. Per-call (--by-call) is the honest ceiling. |
The verb we don’t ship is the point
There is no
gate, approve, or certify verb. Activating a
gate is a host YAML edit you own (metric_status: gating,
threshold_approval: approved), guided by the promoting-a-gate skill — the
agent can type it at your direction; it can never grant the authority. The absence of the verb
is the identity. The honesty charter →