Security
What we protect, and how to tell us when we slip.
EvalGlass Core runs on your machine and in your CI. It reads your repo's evidence and writes a Scorecard. For Core, nothing leaves your repo and there is no hosted runtime to attack. The commercial products — Discovery and Intelligence — have their own boundaries, stated per product below. This page mirrors SECURITY.md.
Data handling — Core
EvalGlass Core is local-first by construction. On Core's required path there is no telemetry, no phone-home, no provider keys, and no hosted service.
- Runs where your code runs. Entirely on your machine and in CI. Your inputs, outputs, and validated reference data never leave your repo.
- No telemetry, no phone-home. The Core runtime does not call out. It reads host-owned evidence and writes typed artifacts locally.
- No keys, no hosted runtime. Core's required paths need no provider SDKs and no SaaS. Nothing to provision, nothing to leak.
- External lanes are opt-in. Optional lanes that touch external services are opt-in, pinned, and deletable. No required import loads them. Extensions →
Security by product
The claims above describe Core. Discovery and Intelligence are separate, pre-alpha commercial products with their own boundaries — so security is stated per product, never as one company-wide promise. Two principles hold across the family, by design:
- No standing combined access. No product holds broad, always-on access across your repo, traces, and models. Access is scoped to what a customer invokes, when they invoke it.
- Separate adapter identities. The repo, trace, and model adapters use separate, least-privilege identities — never one credential that can reach everything.
Each product states its own boundary below. Core’s row is the set of guarantees that already holds today; the Discovery and Intelligence columns describe the intended private-preview boundaries of the products — what they are built to guarantee, not a shipped deployment.
| Boundary | Coreopen source | Intelligenceprivate preview | Discoveryprivate preview |
|---|---|---|---|
| Where it runs | On your machine and in CI. There is no hosted runtime. | Customer-invoked or scheduled — local, your own cloud (VPC), or a bounded private environment. | Customer-invoked under a scoped, time-bounded engagement — local, your own cloud (VPC), or a bounded private environment. No standing service watches your application. |
| What it reads | The repo evidence you point it at — inputs, outputs and validated reference data. | The specific failing runs and the execution evidence you select for a case — bounded to the investigation, no broad standing access. | A bounded representation of your application that you select — code, prompts, tools and selected traces — for the scope you open. No broad or standing access to the rest. |
| What leaves | Nothing leaves your repo. No telemetry, no provider key on the required path. | Only what an explicit model-egress manifest declares. No customer data trains a shared model without opt-in. | The proposals it produces — an application map, coverage gaps, and candidate scenario and rubric evaluations — exported as portable Core artifacts you own. No customer data trains a shared model without opt-in. |
| Identity & access | A fresh install grants no authority; no keys, no combined access. | Separate, least-privilege identities scoped to the case; controlled intervention and replay run in a bounded environment, never against production. | No standing identity. Scoped, time-bounded access is granted per engagement and revoked when it closes; separate least-privilege adapters, never one combined credential. |
| Who decides | Core proposes and measures — the host approves what gates. | Intelligence diagnoses cause through causal replay and controlled intervention under bounded access — an authorized human confirms the cause and approves any change. | Discovery proposes; the customer reviews, edits and accepts — a candidate eval is not an active eval until an authorized human accepts it. |
The product that creates evidence cannot grant itself the authority to decide.
That single rule connects Core’s trust model to the whole family: Core measures, Discovery finds, Intelligence explains — and in every case an authorized human, not the product, makes the consequential organizational decision. No product holds standing combined access, and none is shown running here. The trust model →
Report a vulnerability
Please report privately first. Do not open a public issue for a security problem.
- Preferred: GitHub private reporting. Use GitHub's private vulnerability reporting on the repository. SECURITY.md →
- Fallback: email. Write to contact@evalglass.com with the subject "EvalGlass security".
- Acknowledgement. We aim to acknowledge within 5 working days.
- Disclosure. We follow 90-day coordinated disclosure.
Best effort, no SLA.
EvalGlass is volunteer-maintained and pre-alpha. We will do our best, but we do not promise a response-time SLA. Frame your expectations accordingly.
Scope
| In scope | Out of scope |
|---|---|
The runtime (src/evalglass/**) | Third-party host code that uses EvalGlass |
| Skill-scaffolded code | Disabled optional adapters |
| CI workflows that consume privileged tokens |
One scope note worth stating plainly: a green Scorecard misread as proof of correctness is itself treated as a trust and security issue. Defeating the no-false-confidence guarantee — making an informational, uncalibrated, non-comparable, or partly blocked run look like proof — is in scope. Claim boundary →
Supply chain & supported versions
- Vendored source you can read. The runtime is vendored into your repo, not hidden behind a binary. You can audit what runs.
- Pinned required deps. Required dependencies are pinned, with a manifest and a lock file recording what was vendored.
- Optional deps isolated. Optional-lane dependencies are isolated and pinned, and no required path imports them.
- Supported version. Pre-alpha, version 0.2.1 (tagged) — only
mainis supported; older states are not maintained.
Related