Learn
Project-fitted AI evaluation means checks derived from your own system — your code, output schemas, prompts, and real traces — that your coding agent authors and runs, while human-reviewed records in your repo keep authority over what passes. In practice this is what coding-agent-operated AI quality control looks like.
The guides below are a guided path: read them in order for the full picture, or jump to the stage that matches your question. Prefer to see it work first? Install the plugin, run the quickstart, and learn what to say to your coding agent.
Every guide here preserves the same model: the coding agent operates the evaluation work, and your repo keeps authority over what it means.
01 · authority
Human / host ownerOwns product intent, validation, thresholds, calibration, baselines, and gates. The agent never takes this over.
02 · operator
Coding agentInspects the repo, scaffolds checks, runs EvalGlass, compares results, and explains scorecards. It operates — it does not authorize.
03 · subject
AI application / agentic workflowPerforms behavior across calls, tools, retrieval, steps, trajectories, and sessions — the system under evaluation.
Start at the idea, learn to read the evidence, then evaluate real agentic behavior and ship it with bounded confidence. Each guide is short and stands alone — but in order, they build the whole picture. In a hurry? Jump straight to a goal:
Stage one
The mental model
What agent-operated, project-fitted evaluation is — and the one rule that keeps it honest.
The category, defined: your coding agent operates the evaluation, your repo keeps authority, and the scorecard stays bounded.
Controlled delegation for Claude Code and Codex: what the agent may run, and the authority boundary it can never cross.
The deepest trust idea, made explicit: whatever creates the evidence can never grant the authority to gate on it. Capability is not authority.
Stage two
Read the evidence
The artifact every run returns — and exactly how far a passing result reaches.
How to read the artifact every run returns — values, status, the four verdicts — and what a green one does not license.
Exactly how far a passing result reaches and where it stops: the four filters a claim must survive, and why exit 0 alone is not even a pass.
Stage three
Evaluate agentic behavior
Turn what your app must do into repeatable checks, scored from real traces rather than the model’s self-narration.
Beyond final-answer scoring: the EvalUnit ladder (call, step, trajectory, session), process vs outcome checks, and traces as evidence.
Turn the behavior your app must perform into repeatable evals — examples, metrics, rubrics, thresholds, baselines — and see why generic benchmarks can’t.
The concrete checks — tool choice, arguments, step order, recovery, outcome — scored from observable trace evidence, not self-narrated reasoning.
How exported OTel / OpenInference / local-JSON behavior becomes evidence — imported locally, never a live feed — and why a malformed span is a diagnostic, not a fabricated zero.
When your agent writes your evals: the line between evidence it proposed and reference data a human stood behind — and why generated cases land proposed and can’t gate themselves.
Stage four
Ship with bounded confidence
Put scorecards inside the loop and gate deliberately — host-owned, never automatic.
Put scorecards inside the coding-agent loop so a prompt, model, retriever, or tool change is checked before it becomes unexamined product behavior.
When an opt-in gate is appropriate and why a fresh install gates nothing: from an informational first run to a deliberately activated, host-owned gate.
When Core’s known checks reach their limit and clustering shows a failure mode but not its cause, EvalGlass Intelligence private preview searches for the consequential failures your suite doesn’t yet represent — and turns accepted findings into Core-runnable regressions. Core remains usable on its own.
A coding agent can do a lot of the evaluation work — but the actor or tool that creates the evidence can never grant itself the authority to gate on it. Capability is not authority. That boundary is what keeps an agent-operated evaluation honest, and it runs through every guide here.
Start with the product
