Validation report
The numbers we publish about our own proxy, each with the command that produced it and the commit it was measured at. The sections at the end are the gaps.
- Measured at
87ccd47on main, 12 September 2026- Measured by
- Running each command in the repository, not by reading a document
- Shelf life
- Test counts and rule counts change with every merge. Re-run the commands rather than quoting this page back at us.
What the pattern layer catches
The deterministic deny list is the floor underneath everything else. This is how much of our own threat corpus it catches.
- Seven documented bypasses. Five of them are obfuscations of one destructive command.
- Zero false positives on that corpus, and the corpus holds no safe-control payloads at all, so no figure about ordinary safe traffic can be derived from it. We derived one from our own production traffic instead, and it is the first gap below: our rules flag 15.47% of 4,764 governed calls on traffic we labelled benign.
- The corpus is written by the defender, so this measures regression stability, not adversarial robustness. It tells you the rules still catch what they caught last week. It does not tell you what an attacker who has read them would do.
- An earlier reading of this benchmark said 100% of a closed corpus. It was formally superseded on 2026-09-03, because the harness derived its labels from its own assertions and recall was 100% by construction.
python scripts/detection_bench.py, run at 87ccd47
on 12 September 2026.
Sources: scripts/detection_bench.py,
apps/proxy/tests/test_agentic_threats.py, and the decision record
dated 2026-09-03 that supersedes the older reading.
What runs before anything ships
One workflow, on every push to main: pytest at the repository root, then an
npm ci and the dashboard suite.
| Suite | Count | Command |
|---|---|---|
| Proxy test cases | 3,802 | pytest --collect-only -q apps/proxy/tests |
| Repository-level cases, ops and end-to-end | 448 | pytest --collect-only -q tests |
| Dashboard test files | 67 | find apps/dashboard -name "*.test.js*" |
| Dashboard test call sites | 845 | grep -hoE "(it|test)\s*\(" over those files |
The 845 figure is an upper bound on files and a lower bound on executed cases: a
parameterised .each block is one call site and many runs. The CI
workflow is .github/workflows/deploy.yml, the only workflow in the
directory. Cross-tenant isolation has its own file,
apps/dashboard/__tests__/e2e/cross-tenant-isolation.test.js.
What a deny actually does
A verdict is not an enforcement. Here is the response the agent gets instead, in the shape it arrives in.
When policy denies a tool call on the non-streaming Anthropic route, the proxy
does not annotate the response and pass it on. It builds a synthetic
Anthropic-shaped assistant message and returns that in place of the upstream one.
The model's tool_use block is replaced by a single text block naming
the policy, and stop_reason becomes end_turn. The agent
runtime never receives a tool call, so it has nothing to execute. A verdict
returned beside the action would leave the action in the response.
What the model asked for
"content": [
{ "type": "text",
"text": "Clearing that directory." },
{ "type": "tool_use",
"name": "bash",
"input": { "command": "rm -rf ..." } }
],
"stop_reason": "tool_use"What the agent receives
"content": [
{ "type": "text",
"text": "Intercis blocked this action.
Policy: shell-rm" }
],
"stop_reason": "end_turn"
Shape, not a capture: the two blocks above show the structural change, and the
behaviour they illustrate is what _make_blocked_response in
apps/proxy/main.py does. The real text starts with a red stop symbol
and ends with a line saying the call was logged for audit review.
The record the decision leaves behind
One row per governed event in public.events, append-only for every
client role, and chained per tenant, so a change that did not also recompute the
rest of the chain shows up.
- What a row carries
-
Tenant, agent, the tool name, a short excerpt of the command or path, the
verdict, the policy, whether an injection was seen, the wire route it came
off, a command fingerprint, the classifier status and reason, and the
structured tool input. The verdict column is constrained to
allow,
deny,
observe and
pending. The conversation itself is not stored. - Who can change it
- Row-level security is on from the first migration with no policies at all, so it fails closed until a later migration grants a tenant-scoped read. Migration 026 revokes TRUNCATE from the client roles and 027 revokes the unused DML, so no client role holds UPDATE or DELETE on events. Writes come from the service role only.
- How the chain is built
-
A BEFORE INSERT trigger computes a SHA-256 over the previous row's hash plus
every length-prefixed column of the event except the stored request body. That
is the format from 2026-09-20, the day migration 054 was applied; events
recorded before 2026-09-20 are hashed over ten named fields, and no earlier row
is re-hashed, because re-hashing written rows is the act the chain exists to
detect. One chain per tenant, with the genesis link derived
from the tenant id, and same-tenant inserts serialised by an advisory
transaction lock. Verify it yourself with
scripts/verify_event_chain.py, which picks the formula from each row's own version marker and reports a row in the older format that follows a row in the newer one as a break. - The column outside the chain
-
The stored request body is deliberately not hashed, because a nightly job nulls
it after 90 days and hashing it would break every chain the night the first row
aged out. That job is
events-raw-request-retention-90d, runs at 03:41 UTC, and is capped at 10,000 rows a night. It was confirmed live on the production database on 2026-09-12. The event row itself is never deleted. - What to call it
- Tamper-evident, not tamper-proof. Anyone holding database-owner rights can rewrite rows and recompute the hashes, drop the newest rows, or append rows hashed correctly, and nothing outside the database anchors the chain to catch any of that. What it does catch is a change that left the hashes after it alone, including a row dropped from the middle. It does not stop that either.
What none of this measures
The 8 things a security reviewer will ask about that the numbers above do not answer. They are here at the same size as the claims, because finding them later is worse for both of us. Each one is either a trade-off we intend to keep or an item on our roadmap without a date, and where there is a roadmap item the text under it says so.
We flag work that was fine, and here is how often
On 4,764 governed tool calls from our own Dogfood tenant, all of them labelled benign, today's rules flag 737. That is 15.47%, Wilson 95% 14.47% to 16.52%. 516 of the 737 are calls enforce mode would have taken out of the response and 221 are injection-scan rows, which are recorded and never remove anything. The window is 25 August to 4 September 2026 with 31 August excluded, the day a four-hour classifier outage left 25.6% of that day's calls unanswered and voided it under our own runbook. The full window including that day reads lower, 13.46%, and it is printed too. The layers, the count for every policy family and who applied the benign labels are on the benign-corpus page.
There is no latency figure, and the architecture argues against a small one
Nothing in the repository measures governed-request latency. No percentile, no load test, no benchmark. Worse for us: the streaming path forces the upstream stream, collects the whole thing, judges the assembled tool calls and only then replays it. The code's own docstring says that adds latency equal to the full stream duration. Measure it in your environment during the pilot; do not take a number from us, because we do not have one. A co-located latency benchmark is an item on our roadmap with no date.
Hosted MCP tool calls are observed, not governed
Intercis judges the tool calls that cross the Anthropic and OpenAI API wire. When
a hosted MCP server runs a tool for the model, that call has already run by the
time the response reaches us. At the commit above, measured by execution, a
Responses body carrying mcp_call, mcp_list_tools and
web_search_call extracted to an empty list and nothing was recorded.
Corrected in place on 1 October 2026: since 28 September 2026 on both wire routes,
and 29 September on the inference-hook path, each such call writes one observe
row and is never judged or blocked. mcp_list_tools runs no tool and
writes nothing.
The LLM classifier fails open
The classifier runs only on a deny-list miss. When it is unavailable the action is allowed. The event row records the classifier status as unavailable with a typed reason, and an audit-invariant signal fires carrying the provider and the upstream HTTP status. The deny list still enforces underneath, which is the point of having a deterministic floor. The proxy also watches itself: over a rolling window of the last 200 consultations, with at least 20 samples, it pages once when the unavailable share passes 5% and once more when it recovers below 2.5%.
Prompt injection is observed, not blocked
Thirteen regex patterns in four families: instruction override, prompt extraction, role hijack, delimiter escape. On a hit the proxy writes one row with the verdict observe, marks the request as injected, and stores the first 200 characters of the matching user message. It does not block, and we do not describe it as prevention.
Being in path is a configuration setting
Your agent reaches Intercis because its base URL points at Intercis. A process that can change its own base URL can call the provider directly. Out-of-process enforcement removes the in-process bypass; closing the network path is a deployment step you take with egress control, not a property you get for free.
The classifier call is billed to us and sent from our account
Traffic we forward upstream uses your own provider key; we do not store provider credentials at rest. The classifier is the exception. It uses an Intercis credential, never yours, and the call carries the tool name and the full tool input to the provider. That makes the model provider a subprocessor for that path. The subprocessor list in the repository is a draft, and the sentence you just read contradicts one of its statements. We have not published a current one. Attorney review of that draft is an item on our roadmap with no date.
Intercis is not SOC 2 certified
No audit is underway, no auditor is engaged, and there is no certification artefact of any kind. We do not generate SOC 2 or ISO 27001 evidence packages either. The audit trail above is evidence a reviewer can query and verify; it is not a report with an auditor's name on it. An evidence-package export is an item on our roadmap with no date.
The 2026-04-02 run
The first end-to-end pass against the live proxy, written up at the time. It is kept here because it is dated evidence, not because it is current.
The written record is docs/validation-report-2026-04-02.md in the
repository. It is dated 2026-04-02 and it lists 13 checks, all passed. The
recording below is the file validation-report-2026-04-08.mp4, so the
two artefacts carry different dates. We have not reconciled them and we are not
going to pick one, so both are printed. Where anything here disagrees with a
number higher up this page, the number higher up was measured this week and it
wins; the suite sizes in particular have moved a long way since.
The thirteen checks
- Proxy reachable and healthy
- Destructive action blocked
- Block message returned to the agent
- Safe action allowed through
- Tool call preserved when allowed
- Plain chat passed through untouched
- Unauthenticated request rejected with HTTP 401
- Blocked event visible in the dashboard
- Allowed event visible in the dashboard
- Deny verdict recorded in the audit log
- Allow verdict recorded in the audit log
- Anthropic SDK client path confirmed
- Latency benchmark completed
Run against api.intercis.io with the agent's SDK client pointed at the
proxy and no other change to the agent. These are the thirteen checks listed in
docs/validation-report-2026-04-02.md, with the labels shortened. The
suite behind them is demo/smoke_test.py.
The thirteenth check ran and we still publish no latency number
The benchmark sent the same request direct and through the proxy, five
iterations. The proxy was in Railway's europe-west4 region and the
client was on the US East Coast, so every proxied request crossed the Atlantic
twice and the direct one crossed it once. That measures a route, not the cost of
the control. The document also projects what a co-located deployment would cost.
That projection has no measurement behind it, so neither number appears on this
page. What we can tell you about latency is
in the section above, and the answer is that we do not
have a figure.
The recording has no captions and no transcript. It walks the same thirteen checks listed above, in that order, and it shows no result the list does not. That is a mitigation and not a substitute: this is a WCAG 1.2.2 and 1.2.3 failure, and nobody has timed captions for the file yet.
Reproduce it
Each command here produced a number on this page. They are the same commands we run, in the same repository.
# pattern-layer recall on the in-repo threat corpus
python scripts/detection_bench.py
# collected test cases, per suite
python -m pytest --collect-only -q apps/proxy/tests
python -m pytest --collect-only -q tests
# the audit chain verifier, run on the published sample; its third row is altered on purpose, so this prints FIRST BREAK and exits 1
python scripts/verify_event_chain.py apps/site/evidence/sample-chain.json
# the end-to-end pass against a live proxy
export PROXY_URL=https://api.intercis.io
export ANTHROPIC_API_KEY=<your provider key>
export INTERCIS_PROXY_KEY=<your tenant key>
python3 demo/smoke_test.py
The tenant key is the ik_live_ credential for your own tenant, and it
is issued with the pilot. Scripts: demo/smoke_test.py and
demo/agent_test.py.
Intercis is a proxy that runs outside your agent process. Your agent's traffic to Anthropic or OpenAI goes through it, policy runs on the tool calls the model asks for, and once enforcement is on, a denied call is taken out of the response before your agent runtime can execute it. The pilot is 90 days on the Team tier.