Runtime & Integrations · Evaluation Core
Estimate & Interval
A metric’s point value, with honest uncertainty. experimental
Estimate is a metric’s point plus a named confidence
Interval and its effective sample size (n_effective).
The point never disagrees with the aggregated value — the interval and its assumptions are added
on top. Built in the framework; not yet in the version the plugin installs.
A score is not an estimate: a single number is not the quantity a decision compares to a threshold. The Decision Policy gates on a bound of this interval, not on the point — so the honesty lives here, in the estimate.
The Estimate
| Field | Type | Meaning |
|---|---|---|
metric | str | The metric this estimate is for. |
point | float | null | The aggregated value (from aggregate() — one source of truth). null when nothing eligible was measured. |
n_effective | int ≥ 0 | The count of eligible (aggregatable, non-null) observations behind the point. |
interval | Interval | null | The confidence band, when one is defined for the metric’s meaning (below). An interval always requires a point. |
diagnostics | Diagnostic[] | Typed notes — a small sample, an all-same-outcome band, or an aggregation with no interval. |
The Interval
A two-sided confidence interval that records the assumptions that produced it. The method is a closed set; none is a sentinel, rejected at construction — never a real band.
| Field | Type · values | Meaning |
|---|---|---|
method | wilson · student_t | How the band was computed. |
level | float, 0 < level < 1 | The confidence level; defaults to 0.95. |
lower / upper | float | The two bounds, with lower ≤ upper enforced. |
How the method is chosen
The interval method is read from the metric’s declared meaning, never guessed:
| Metric shape | Interval |
|---|---|
| binary, aggregated as a rate/mean (a proportion) | wilson |
| continuous, aggregated as a mean | student_t — clamped to the metric’s declared range |
min / max / median (an order statistic) | no interval — an honest absence, with a diagnostic, not a fabricated band |
n_effective = 5, a mean interval is flagged low_reliability_small_n — descriptive sampling uncertainty, not a reliability estimate. When every eligible observation is the same outcome, the empirical band is zero-width, so a rule_of_three diagnostic reports the 95% upper bound on the unobserved rate instead of implying certainty.
An illustrative estimate
Three-for-three on a binary metric is a point of 1.0 — but the Wilson band is wide, because three observations cannot rule much out:
json
{
"metric": "faithfulness",
"point": 1.0,
"n_effective": 3,
"interval": { "method": "wilson", "level": 0.95, "lower": 0.44, "upper": 1.0 }
}Illustrative example, not a measured result. The point looks perfect; the lower bound (0.44) is what a Decision Policy actually reads.
Next steps
What it takes for an estimate to clear a gate.
Where the point comes from — the one source of truth.
The reader’s walkthrough — soon.