Diagnose & intervene
Build the causal execution graph, form hypotheses, and test candidate causes with controlled replay and intervention — reporting the strongest supported evidence level, never a story.
EvalGlass Intelligence · causal failure intelligence
Understand why it failed, and make the evaluation system improve.
Core runs the evaluations you define; Discovery finds the ones you are missing. Intelligence is the advanced reasoning system for the hard failure — it determines what actually caused it, tests the cause with controlled replay and intervention, verifies the repair, and turns the learning into a durable evaluation you keep.
Cause, not coincidence
A plausible explanation is not a cause.
Discovery can tell you a retry path is untested; Core can tell you a metric moved. Intelligence begins after that — once you have an executable suite and a plausible hypothesis, and you need to know why a difficult, intermittent or consequential failure actually happens. It reasons about cause, and it refuses to compress that into a single unexplained number:
- A causal execution graph. A model deep enough to reason about cause, where edges are candidate dependencies, not mere sequence — across inputs, routing, context, retrieval, model calls, tools, state, approvals, side effects and outcomes.
- Controlled intervention. Prefer experiments where one meaningful factor changes — remove a prompt component, mock a tool response, restrict a permission, substitute a model, override a route — because an intervention is stronger evidence than an explanation.
- A minimal reproducer. The smallest scenario that still preserves the failure, runnable in Core or your test harness, with an exact reproduction command and provenance back to the incident.
- Verified repair. Intelligence proposes; you approve the change. It then checks the failure is gone, expected behavior remains, nearby variants do not regress, and the new evaluation detects the original behavior.
Graded evidence
Association is not causation — every conclusion states its level.
A conclusion is never presented as more certain than the evidence supports. Each causal claim carries an explicit evidence level, and a graph edge moves up the ladder only as evidence improves:
- Observed association. The failure and a candidate factor co-occur.
- Structural dependency. The application structure supports a dependency.
- Counterfactual support. Changing the factor changes the outcome under controlled conditions.
- Intervention-validated cause. A targeted intervention reliably removes or creates the failure.
- Repair-validated cause. The implemented repair removes the failure and the regression test stays stable.
- Human-confirmed. A human reviewer confirms the cause — human confirmation stays part of the causal record.
Uncertainty and failed interventions are valuable outputs, not failures: a non-discriminating experiment narrows the hypothesis space and is kept.
The method
Diagnose. Intervene. Verify. Learn.
Verify the repair
Minimize the incident to a reproducer, and once you approve a change, verify the failure is gone with no nearby regressions — a repair-validated cause is the strongest evidence of all.
Learn
Feed the Evaluation Learning Loop: every incident and change becomes a proposed new or stronger evaluation you promote — so the suite gets better after every important failure. Everything durable runs in Core.
The deep layer — without false confidence.
Intelligence improves your evaluations, not your production system autonomously; humans confirm the cause and approve every change, and no proposal enters the active suite by itself. It never calls your AI safe — only what the evidence supports, at a stated level. It exports to your Jira, incident, GRC and audit systems but does not become those systems. The work is bounded and customer-controlled — your application, your evidence window, your environment — with no broad standing access, and every durable artifact stays runnable in open-source Core. Investigate a hard failure →
How you start
A fixed-scope Intelligence Investigation.
Intelligence is a proprietary product, private preview. It begins with an Intelligence Investigation: one hard failure or incident on one application, bounded evidence, a hypothesis set, at least one controlled intervention where feasible, a minimal reproducer, a verified repair or materially narrowed uncertainty, and a durable Core regression artifact. Ongoing work is scoped by covered applications, diagnostic capacity and deployment, may include Discovery for those applications, and offers private or sovereign deployment. Pricing is agreed privately in the preview and is not published; there are no list-price numbers here yet.
Related
EvalGlass Core
the open runtime every durable artifact runs in
EvalGlass Discovery
finds the missing evals and the hypotheses to investigate
The product family
Core executes, Discovery finds, Intelligence explains