Run and read
Confidence & intervals
Why a perfect score can still fail to gate.
n_effective) — and the default gate decides on the lower bound, not the
point. So a lucky 3/3 does not pass; a run decides on the point only under a named
point-smoke policy, a deliberate and recorded choice.
No false green is built on this. A number with no sample size and no interval is not evidence — it is an anecdote with a decimal point. This guide is the intuition; the typed contract lives in the reference pages linked throughout.
A number is not evidence
Three correct answers out of three is a point of 1.0 — but three observations rule almost nothing
out. The honest form of that result carries its own uncertainty: a point, its 95% interval, and the
count behind it (n_effective). Read the full contract on
Estimate & Interval.
The gate reads the lower bound
A Decision Policy compares the lower confidence bound (LCB) to the threshold, not the hopeful point — for a higher-is-better metric; the upper bound for lower-is-better. It reads the edge of the interval that could hurt you.
The escape hatch is explicit and named: a policy that sets decision_statistic: point — a
point-smoke check for dev — decides on the point instead. It is a deliberate,
recorded choice, never the silent default.
A fake judge can never gate
Authority is capability-typed: an uncalibrated or fake judge produces evidence, not
authority, so it can never move a gate. A real one earns calibrated only through a
computed, self-proving agreement study and a human approval bound to the exact rig. The full story is
Judge Integrity.
point-smoke policy
(decision_statistic: point) — a deliberate, recorded choice for dev, never the silent
default.