Trust Model
Diagnostic Clusters
Group a run’s failures by cause. next
Scorecard.clusters is an additive field that groups a run’s
failing and non-scored items by their shared Diagnostic.code,
per metric — turning a flat faithfulness = 0.82 into an actionable
failure mode (“the 18% that failed are all missing-citation cases”). It adds explanatory
structure only: it carries no value, changes no verdict, and manufactures no authority. Built in the
framework; not yet in the version the plugin installs.
A metric value says that something is wrong, not what. Clusters answer “which
items failed, and do they share a cause?” by grouping by explicit diagnostic code —
never by list order. This is a different axis from the call-identity metrics explorer:
clusters group failure instances by cause; the explorer would map a score to its source call
site. Those are kept separate (see below).
The DiagnosticCluster
A cluster is one failure mode: the items of metric whose diagnostics share
code. Each score contributes once per code (duplicate codes on one score
count the item once). It carries no score value — a
blocked / non_evaluable /
error item is grouped by its cause and counted, never
coerced to 0.0.
| Field | Type | Meaning |
|---|---|---|
metric | str | The metric these failing/non-scored items belong to. |
code | str | The shared Diagnostic.code — the cause the items have in common. |
severity | info · warning · error | The representative severity (the code’s highest, then least message). |
count | int ≥ 0 | How many items (scores) carried this code — not a value. |
message | str | The representative message for the code. |
Order-invariant by design
The pure, stdlib-only cluster(scores) function groups by (metric, code) and
sorts the result canonically — severity descending, then count descending, then (metric, code).
Two runs over the same scores in any order produce an equal list. That
order-invariance is what lets the engine store the clusters on the Scorecard and still have them
recomputed exactly on load.
example_ids: it names the failure
mode (cause + count + severity). Drilling down to which items is the call-identity
job of view --by-call — kept off the cluster so the recompute stays robust when a
score’s identity is absent (an older artifact).
The recompute guard
Scorecard.clusters is emitted only when non-empty, so a pre-P3 scorecard.json
stays byte-identical and is parsed absent→[]. On load, _verify_consistency
recomputes the clusters from the persisted scores and rejects any mismatch: a
fabricated or hand-edited cluster fails closed, exactly like a tampered aggregate. A record with no
clusters is not checked (the additive-field convention) — dropping the clusters loses explanatory
structure only, never a value or a verdict.
report.md (a “Failure clusters”
section) and report.html (<code> · N cases per metric) from the typed
field only — no re-grouping, no authority. scorecard.json stays the one source of truth.
A different axis from the explorer
Diagnostic clusters group failure instances by cause (next). They are a
different axis from two things they are easily confused with, and neither is badged the same way:
| Construct | Groups by | Status |
|---|---|---|
| Diagnostic clusters | shared Diagnostic.code (failure mode) | next |
| Call-identity “metrics explorer” | a score’s source call site / call identity | unbuilt |
| Per-source-function view | the discovered source function that produced the call | non-goal |
In one sentence: clusters group failures by Diagnostic.code and are next; the
call-identity metrics explorer would map a score to its source call site and is unbuilt; the
per-source-function view (ADR 0037) is a deliberate non-goal,
never “planned.”
An illustrative cluster
Two of eleven faithfulness items failed for the same reason — grouped by cause, counted, and carrying no value:
json
{
"metric": "faithfulness",
"code": "missing_citation",
"severity": "error",
"count": 2,
"message": "Claim not supported by any retrieved passage."
}Illustrative example, not a measured result. The cluster tells you the shared cause and how many items hit it; it does not change any score and it does not gate. A run showing clusters is still informational unless a metric is authorised to gate.