Trust Model
Judge Integrity
A judge earns authority — it never asserts it. experimental
JudgeInstrument (identity), a self-proving
JudgeAgreementStudy (agreement with human gold), a digest-bound
AuthorityGrant (approval tied to what it approved), and an optional
ClaimSpec (the validity argument). Built in the framework; not yet in the version the
plugin installs.
Alpha let authority rest on a bare "approved" token and a declared
status: calibrated. The epistemic core replaces every assertion with something the
core re-checks — so a judge that changed, a statistic that was typed in, or an approval whose rig
moved underneath it can never keep its gate. A
fake or uncalibrated judge stays informational.
JudgeInstrument — content-addressed identity
The complete identity of the measuring instrument — not opaque *_ref names, but the content digests of the resolved prompt and rubric. Any real change shifts the digest.
| Field | Type |
|---|---|
provider · model | str · str |
prompt_sha256 · rubric_sha256 | 64-char hex — the content of the resolved prompt / rubric, not their names |
parser_version | str |
seed · temperature · decoding | int | null · float | null · map |
digest() content-addresses every field. Swap the provider, the seed, or the resolved prompt text and the digest moves — so an old calibration bound to it reads drifted instead of silently blessing a new instrument.
JudgeAgreementStudy — computed, self-proving
Agreement with human gold, computed from a 2×2 Confusion table (tp/fp/fn/tn) and recomputed on construction: the stored percent_agreement and Cohen’s kappa must match the table, or the study is rejected. A number cannot be typed in.
| Field | Meaning |
|---|---|
judge_instrument_sha256 · rubric_sha256 | The instrument this study measured — a moved digest means the study no longer applies. |
confusion | The 2×2 table the statistics are re-derived from. |
percent_agreement · kappa | In [0, 1] and [−1, 1]; both must equal the recomputed values. |
variance_runs · protocol_version | Run count and the study protocol. |
approver · rationale | Optional — the study is not authority on its own. |
A study earns calibrated only with a human approver and a current, matching instrument — never by arithmetic alone.
AuthorityGrant — an approval bound to what it approved
A human approval carrying the SHA-256 of every artifact it blessed — the decision policy, the dataset validation, the evaluator capability, the study. The core re-verifies each against the current run.
mismatched and the gate falls back to informational — never earned authority for this rig — rather than passing. Grant status is matched · mismatched · expired · missing. The tool never writes a grant; a human authors and approves it.
ClaimSpec — the validity argument
An optional, host-owned argument for what a number means: its construct, intended_use, target_population, and sampling_frame, plus known_threats, excluded_constructs, prohibited_extrapolations, and validity_evidence_refs. It carries no gating power on its own — but the scorecard’s claim is narrower without it, and its digest can be bound by a grant.
ClaimSpec is where a human states, and a reviewer checks, what the number may be read to mean.