Trust Model
Input Convergence
How different inputs become one evaluator shape.
Every way evidence enters EvalGlass — a JSONL dataset, a recorded trace, an open-convention
trace, a subprocess replay — normalizes to one spine: TraceEnvelope → EvalUnit →
Example. An evaluator only ever sees a plain Example; vendor and
trace shapes are dropped at the edge.
Hosts capture LLM behavior in many formats. If each format reached the evaluators directly, every metric would have to understand every vendor shape, and a vendor object could smuggle meaning — or an effect — into the Core. Convergence removes that: the Runtime Harness converts each source at the boundary, and the Evaluation Core sees one normalized item.
INPUTS
many shapes · host-supplied
JSONL dataset
local trace
open-convention trace
subprocess replay
normalize at the edge
RUNTIME HARNESS EDGE
one spine · vendor objects dropped
TraceEnvelope
EvalUnit
Example
✗vendor & trace objects dropped
only an Example
EVALUATION CORE
effect-free · sees only an Example
Evaluator
- inputs & Harness edge
- effect-free Core
Example at the Runtime Harness edge; vendor and raw trace objects are dropped there. Only an Example crosses into the effect-free Core — many shapes in, one shape out.The three spine types
| Stage | What it is | What it carries |
|---|---|---|
TraceEnvelope | Vendor-neutral normalized host behavior. | trace_id, source, behavior, metadata, and a required data policy. |
EvalUnit | The declared slice of behavior to evaluate. | unit_id, kind (call in the MVP), trace_id, members. |
Example | The evaluator-ready item. | input, output, its unit, an optional reference, context, metadata, provenance. |
A dataset route fills the reference (the gold answer); a trace route fills the
recorded output. Both arrive as the same Example, so a metric never
needs to know which route produced it.
Convergence is a trust boundary
Two properties fall out of normalizing at the edge:
- Evaluators receive data, not adapters. The Core cannot reach through an
Exampleto read a file or call a model, so a metric cannot quietly perform an effect. - Incomplete input cannot pass a gate. A malformed record does not vanish — the Harness turns it into a run-level route-error example that scores error, so a broken source surfaces instead of silently shrinking the run.
New sources never touch the Core
Adding a backend means implementing a port (typically
TraceSource) that emits TraceEnvelopes. The rest of the spine is
unchanged, and optional backends arrive as deletable lanes that drop their vendor wrapper at the
edge — so no vendor object ever reaches the Core, the RunRecord, or the
Scorecard.