Threat model
The adversary is false confidence.
Not a malicious outsider — the well-meaning agent, score, or report that would quietly claim more trust than a run actually earned. Here is how each path to that overclaim is closed by construction.
The threat actor
There is no attacker breaking in. The threat is the eager helper.
It is the coding agent that would approve its own gate to make the build green. It is the AI judge that grades on vibes and calls it measurement. It is the report sentence that sounds like proof. It is the regression flagged against runs that were never comparable. Each of these inflates trust with good intentions — and each is the thing EvalGlass is built to refuse. The defenses below are not policies you opt into; they are how the core is shaped.
Attack → defense
Every row is a way trust gets inflated, and the structural reason it cannot land here.
| Attack | How EvalGlass prevents it |
|---|---|
| An agent self-approves a gate to turn the build green. | Only a human-approved threshold can gate. Agent-proposed values stay informational until you validate them. Evidence & approval → |
| An AI judge grades on trust alone and gates on its own opinion. | A judge cannot gate until it is calibrated against host labels. Uncalibrated judges stay advisory. Judge calibration → |
| Report prose overclaims what the run proved. | Authority is typed data, and reports render from the Scorecard only. Prose cannot carry a claim the data does not. Authority → |
| A regression is claimed without comparable runs. | Baseline comparability is a typed claim. A delta between non-comparable runs is reported as not comparable, not as a regression — and when the scheduled watcher (evalglass watch) finds none, that is not a clean bill of health: a no-comparable-regression result never means quality is fine. next Baseline comparability → |
| A scorecard record is hand-edited so a failing run reads as passing. | The record is content-addressed and re-derived on load: a load-time recompute rejects a tampered record rather than trusting the altered numbers, and authority is bound to the digest. now Confidence & intervals → |
| The calibrated judge is swapped for a different model or prompt after approval. | A judge is a content-addressed JudgeInstrument: swap it and the instrument reads drifted, so the earlier calibration and its authority do not transfer. now Judge integrity → |
| A perfect score over a handful of examples is used to clear a gate. | The default DecisionPolicy gates on the lower confidence bound, not the point — so a perfect 3/3 (lower bound ≈ 0.44) fails the default gate; it passes only under a named point-smoke policy. now Why a perfect score can fail → |
| Generated reference data or proposed thresholds quietly become authority. | Scaffolded assets start informational and stay that way until you validate them. Generation never confers authority. Evidence & approval → |
| Verdict logic is forked into the CLI or the report. | Only the Verdict Engine decides, and the CI exit derives solely from ci_should_fail. There is no second place a pass can be minted. Verdict Engine → · CI exit codes → |
| An optional lane becomes silently required. | Lanes are isolated, opt-in, and deletable. No required path imports one, and deletion leaves the core tests green. Extension lanes → |
New threats the family introduces
The product family opens new ways false confidence can enter. Discovery and Intelligence are private preview, so these describe the intended structural defense — the shape each product is being built to hold — not a shipped mechanism. The doctrine is unchanged: the product that creates evidence never grants itself the authority to make the decision.
| Attack | How the family is designed to prevent it |
|---|---|
| Discovery presents a hypothesis as a proven finding. | Every Discovery output is labelled a hypothesis and carries an explicit evidence level; a coverage gap or candidate eval never becomes an accepted eval until a person reviews, edits and accepts it. Discovery proposes; the customer accepts. By design; Discovery is in private preview. |
| A draft finding is promoted to an authoritative record without review. | Findings stay draft in a working space until a person accepts them; acceptance is a human act, never automatic. The evidence producer cannot grant its own authority. By design; pre-alpha. |
| Intelligence overclaims a cause. | An Intelligence root-cause record carries a graded evidence level — association is not causation — rising from observed association only as far as a controlled intervention actually supports, and a cause stands only once a person confirms it. By design; Intelligence is in private preview. |
| A commercial service receives broader data than its declared manifest. | Each service declares its inputs and egress up front; access is scoped to what a customer invokes, with separate least-privilege identities and no standing combined access. By design; pre-alpha. |
| A Discovery candidate silently starts gating the build. | Candidates are proposals, not gates: a candidate eval is not an active eval, accepted proposals become inspectable Core artifacts you own, and nothing gates until you decide it does. By design; Discovery is in private preview. |
Local-first as a data defense
The other quiet threat is data leaving your control. Core closes that by not having a place to send it. For Core there is no hosted platform, no account, no provider keys, no telemetry — your evidence, datasets, and verdicts never leave the repo. Everything runs on your machine and in your CI from vendored source, and the runtime keeps working after the install skill is removed. Nothing phones home because Core has no home to phone. (Discovery and Intelligence run under the bounded, customer-controlled model above, not as a hosted service that ingests your repo.)
Out of scope — loudly.
EvalGlass defends against false confidence in evaluation. It does not assert that your model is safe, it does not red-team your application, and it does not replace a security review. A green Scorecard is an honest statement about the evidence behind one evaluation run — nothing more. This is not a security product, and reading it as one would be exactly the kind of overclaim it exists to prevent.
See it in the contracts
The single decision point is the Verdict Engine. Read how it turns typed authority and score state into one outcome — and why nothing else is allowed to.
Related