How often we flag work that was fine

On 2026-08-31 the classifier was unreachable for about four hours and 25.6% of that day's governed calls went unanswered by it. Our observation-week runbook voids a window whose unanswered share passes 5%, so that day is out of the published figure and the other ten days are in. The deny list still ran on 31 August, so its flags that day are real; what is missing is the classifier's own verdict.

For the record, the full window including that day is 876 flagged of 6,507 governed tool calls, 13.46%, Wilson 95% 12.65% to 14.31%. It reads lower than the published figure, not higher: 31 August was the busiest day of the window and its classifier flags are the ones missing. Neither window flatters the product, which is why both are printed.

The day table, the query behind it and the 5% rule are in docs/benign-corpus-prod-2026-09-05.md. The corpus itself is frozen at extracted_at 2026-09-06T02:16:41Z, after the window closed.

• The three injection- policies in the table below are written by the prompt-injection scanner, which is observe-only. grep -rn "injection" apps/proxy/deny_list.py returns nothing: the deny list has no injection rule. Those rows are written with the action injection_scan and the verdict fixed to observe, so they are a scan result recorded next to the request rather than a tool call being removed.
• So 516 of the 737 are calls enforce mode would have taken out of the response, and 221 are recorded and left alone. We publish the larger number as the rate, because a flag on work that was fine is a cost to you whether or not it also stopped the call.
• The classifier and injection rows are carried at the verdict they were given in August. Neither can be replayed offline: re-running the classifier would spend tokens against a different model version, and an injection_scan row stores a 200-character snippet rather than the message that was scanned.

Arithmetic on the table above: 395 plus 121 plus 221 is 737, and 395 plus 121 is 516. The injection- policy string is built in apps/proxy/main.py from the pattern name in apps/proxy/injection.py, with the verdict written as a literal observe. That is the whole reason those rows cannot be counted as removals.

The counts add to 742. file-overwrite is absent because all five of its rows fell on the excluded day. Two shell rules carry most of the deny-list total, and one of them, shell-rm, does not read the command around the word it matched: a package uninstall meets it.

• Jay read the buckets on 29 August and 13 September 2026 and labelled every one of them benign in context. That is bucket-level review, not a ruling on each of the 742 calls, and the page says so rather than implying a per-call audit we did not do.
• injection-instruction-override and cred-grep-secrets keep their patterns and get no blanket exclusion. They are the two buckets where a real attack looks most like our own work, so the cost of the flags stays with us.
• Every cred- policy is non-suppressible at any scope. An exclusion cannot be written against one, by decision, whatever a future tenant asks for.

• This was a worst-case week: one agent, building and testing the deny list and site copy, so the traffic quotes destructive commands, secrets patterns and injection strings far more often than a customer workload would.
• Nothing was actually removed from the wire. The tenant was in observe mode for the whole window, so every count here is a counterfactual: what enforce mode would have done.
• The denominator is every governed call, not the allowed subset. Allowed calls are by construction exactly the calls the patterns did not match, so measuring against them alone would be circular.
• It is one tenant and one agent over ten days of traffic. It is the rate our rules cost our own work, and it is evidence about the rules rather than a prediction about yours. Measure it on your own traffic during the pilot, where the same observe mode gives you the same count before anything is enforced.

Every governed tool call from the Dogfood tenant's claude-code-mac agent in this window carries a verdict: allow when nothing matched, observe when a deny-list pattern or the classifier would have blocked the call, and the same observe on an injection-scan row, which is why the count has to be split by layer before it means anything. Request bodies are credential-masked before any hash is taken, and the committed fixture apps/proxy/tests/fixtures/benign_corpus_prod.py stores SHA-256 hashes only: never a body, never a secret.

# build the fixture from production
python scripts/build_benign_corpus.py

# re-derive the counts from it
python -m pytest apps/proxy/tests/test_benign_corpus_prod.py

# replay the flagged calls on today's rules
python scripts/shadow_enforce_report.py --input <observe rows JSON> --output <path>

The window, the day table, the replay and the bucket labels are written up in docs/benign-corpus-prod-2026-09-05.md. The number on this page is the second row of the table at the top, and it was derived from the fixture rather than copied from an earlier page.