Start here

Quickstart

Install the plugin, then just talk to your coding agent.

EvalGlass is agentic-native: you operate it entirely through your coding agent — Claude Code or Codex — in plain language. No CLI to memorize, no keys, nothing runs on a server. This is the fastest path from install to a first, honest Scorecard.

Five things to say, in order. The first result is not a green “pass” — it is an informational Scorecard, by design. That is exactly the right first thing to see: EvalGlass measuring honestly and claiming only what the evidence supports.

1 · Install the plugin

One time, in your coding agent — add the marketplace, then the plugin:

in your coding agent

/plugin marketplace add Evalglass/evalglass-core
/plugin install evalglass-core@evalglass

No keys, no telemetry. Codex is supported as a second runtime. For more detail — and the direct, non-agent CLI if you prefer it — see Get the plugin.

Want to see it work first? (about 10 seconds) Before touching your repo, ask your agent to “Run the EvalGlass quickstart.” It renders a real informational Scorecard from a bundled sample — no setup, no keys — so you can read a result before adopting anything. (/evalglass run --example quickstart)

2 · Set up your app

say to your agent

Set up EvalGlass for my app.

A read-only, consent-gated scan finds candidate LLM call sites, vendors the runtime into your repo as plain code you can read, and scaffolds a proposed dataset, metrics, and CI. Nothing gates yet — it is a reviewed-by-you starting point, never a promise it found every call. (/evalglass setup)

3 · Bring your behavior in

say to your agent

Connect our exported traces.

Your exported OpenTelemetry / OpenInference or local trace JSON is imported — no SDK, no network — so checks run over how your app actually behaved. (/evalglass connect)

4 · Choose what to measure

say to your agent

Add a metric for the failure I care about.

In Core you author the checks your app needs — explicitly, as owned config — or start from the proposed set step 2 scaffolded. Deriving what to measure by deep-reading your code, output schemas, prompts and traces into a proposed MetricCatalog anchored to how your app fails is EvalGlass Discovery, a separate product (private preview), not part of the open-source Core plugin. Whichever route, everything lands proposed / informational. (/evalglass add-metric; deep-read discovery → EvalGlass Discovery)

You’re not blocked without Discovery Step 2 already scaffolds a proposed starting set of metrics, and you can author any check explicitly in Core today. Automated, application-specific metric discovery is EvalGlass Discovery — a separate product (private preview), not a Core release. How discovery works →

5 · Run & read

say to your agent

Run the evaluation and show me the Scorecard.

You get a Scorecard with per-metric values, validity, and one honest verdict. A fresh run gates on nothing, so it is informational, not pass — it exits 0 because nothing has been approved to enforce, and a metric that could not be scored is marked non_evaluable, never 0.0. (/evalglass run → view)

Your first result is informational — that’s correct A fresh install that showed “pass” would be manufacturing confidence from data no human reviewed. To make a check actually enforce CI, you validate a dataset and approve a threshold — there is no gate, approve, or certify verb, so that authority is always yours. Read a Scorecard → · Promote a Gate →

Next steps

Operate it

Agent commands

Everything to say, and every verb underneath.

Build checks

Discover metrics

Core scans call sites; deriving what to measure is EvalGlass Discovery.

Run and read

Read a Scorecard

Understand every field in the result you produced.

Ship

Promote a Gate

Turn one informational metric into real enforcement.