EvalGlass vocabulary

Vocabulary

Every canonical term, defined once.

EvalGlass uses a fixed vocabulary so a term means the same thing on every page. This is the single definition source; other pages link here on first use of a proper noun.

Layers

Term	Definition
Evaluation Core	The effect-free centre: contracts, score semantics, aggregation, provenance, authority, and the Verdict Engine. Standard-library only; no I/O.
Runtime Harness	The effectful layer: CLI, config, adapters, replay, judge calls, persistence, reports, and CI-exit mapping. Owns effects, not meaning.
EvalGlass Skill	The integration-time installer skill (discover / plan / install / re-vendor), now delivered as part of the EvalGlass plugin. Grants no authority; not needed at runtime.
Host-owned truth	Datasets, references, rubrics, calibration, approved thresholds, baselines, host evaluators, and domain decisions — everything that confers authority.

Data spine

Term	Definition
TraceEnvelope	Vendor-neutral normalized host behavior plus metadata, data policy, and provenance.
EvalUnit	The declared slice of behavior evaluated: `call`, `step`, `trajectory`, or `session`.
Example	The evaluator-ready item: input, output, unit, optional reference, context, metadata, provenance.
Gold	Host-owned, validated reference data a `reference` metric compares against — never synthetic by default, never self-authorizing; only a host validation makes it gating-grade. (Marketing/trust pages say validated reference data instead.)
EvidenceBundle	References, source material, judge evidence, verifier evidence, runtime errors, and trace fragments passed into the core.

Measurement & meaning

Term	Definition
MetricSpec	A metric’s declared meaning: lens, score type, direction, profile, evidence needs, prerequisites, aggregation, threshold, authority, data policy.
Lens / Metric Type	Whether a metric is `reference` (compares to gold) or `non_reference` (inspects output alone).
Score	One metric result: value, status, validity, diagnostics, evidence refs, evaluator version, provenance.
ScoreBatch	A set of related Scores emitted by one evaluator.
Score subject identity	The optional `example_id`/`unit_id` each Score carries (framework slice F1): which Example/EvalUnit it measured. Additive provenance, not meaning — it lets a reader group scores per call; it is never authority or source-function attribution.
Diagnostic	Structured reason for a non-perfect state: code, severity, message, location, details, cause, evidence refs.

Trust & outcome

Term	Definition
Authority	Typed inputs deciding whether a metric may gate: informational, blocked, or can_gate.
Provenance	Structured run fingerprints recording how a run was produced, across ten dimensions.
Baseline comparability	Whether two runs may be compared: comparable, not_comparable, missing_baseline, comparison_not_requested.
Data policy	Per-source egress state: permitted, redacted, forbidden, missing, unknown — enforced before effects.
Judge	An LLM-as-judge model whose evidence may gate only after host calibration.
Verdict Engine	The single path turning measurement and authority into a run outcome.
RunRecord / Scorecard	The complete machine record of a run / its machine-and-human summary, the primary artifacts.

Integration

Term	Definition
Port / Adapter	A typed effect boundary / its concrete implementation. Adapters return evidence or data, never scores or verdicts.
Extension Lane	An opt-in, deletable integration attached through an existing port.
Vendoring	Installing managed runtime source into a host repo under `evals/_evalglass/`. The skill copies only `core`/`harness`/`adapters`; the runtime then runs independently of the plugin and any agent.

Plugin surface

Term	Definition
`/evalglass <verb>`	The umbrella the plugin exposes: `setup`, `connect`, `run`, `view`, `explain`, `compare`, `baseline`, `ci` (plus the v1.1 authoring verbs `add-metric`/`add-judge`/`calibrate`). Bare `/evalglass` is an honest status overview that runs nothing. There is no gate/approve/certify verb — the absence is the identity.
view granularities	per-metric (default) · per-call (`--by-call`, grouped by Score subject identity, never list order) · per-source-function (advanced, not yet — needs a trace↔call-site correlation that does not exist).
Codex second runtime	EvalGlass also packages for Codex from the same canonical `skills/` tree via a thin manifest; the vendored runtime is identical across runtimes. A public Codex marketplace listing is a maintainer step, not yet published.

Banned terms Never write kernel, test kernel, or pure kernel for the effect-free centre. Use Evaluation Core, effect-free core, or core isolation. Likewise, never describe a result as proven-correct, warranted, or endorsed — a Scorecard is a bounded claim, not a stamp of approval.

Confidence & decisions

Estimate	A metric’s point value plus a confidence `Interval` and its effective sample size — not a bare number.
Interval	A named two-sided confidence band (`wilson` or `student_t`) with a level, lower, and upper.
n_effective	The count of eligible observations behind a point — the sample size a decision weighs.
Lower confidence bound (LCB)	The conservative edge of the interval a default `DecisionPolicy` gates on, so a lucky small sample does not pass.
DecisionPolicy	The host-owned rule turning an Estimate into pass/fail/blocked; gates on the LCB by default; content-addressed.
Point-smoke policy	A named policy that sets `decision_statistic: point` — the explicit dev escape hatch, never the silent default.
JudgeInstrument	The content-addressed identity of a judge (provider, model, prompt/rubric digests); a change reads `drifted`.
JudgeAgreementStudy	A computed, self-proving judge-vs-gold study (Cohen’s kappa recomputed from the confusion table).
AuthorityGrant	A human approval bound to the SHA-256 of what it approved; edit the rig and it resolves `mismatched`.
ClaimSpec	The optional construct/validity argument for a metric’s interpretation; no gating power on its own.

Next steps

Trust Model

Claim Boundary

The term every other term protects.

Contracts Reference

Enums

Every closed enum value.