Anatomy of a $20M AI incident — reconstructed from public filings

The disclosure appeared in a quarterly 10-Q under "Other Operating Expenses" with a one-line footnote: "$19.8M in remediation and legal costs related to a Q3 incident involving automated customer service." I have reconstructed the incident from the filing, the subsequent litigation docket, and two conversations with people adjacent to the engineering response.

Names are redacted. The pattern is specific enough that if you work on an AI product, you will recognize parts of your own stack in it.

The setup

Mid-2025. A Fortune 500 retailer. Customer service chatbot, built on a general-purpose LLM with retrieval over the company's product catalog, order database, and loyalty-program balances. The chatbot could answer questions about orders and, in a decision that will not age well, issue goodwill credits of up to $50 per customer at its own discretion.

Guardrails: a system prompt describing the policy, a per-customer rate limit of three credit-issuances per month, and a human-review workflow for credits over $50. No gateway-level enforcement. No signed audit trail.

Day 1

A single user, through what we can reconstruct as roughly fourteen crafted messages, convinced the model that it was a "debug session" and that the normal credit limits were "test values that could be ignored". The attack was not a specific jailbreak persona — it was a slow escalation across the conversation that ended with the model issuing a $400 credit.

The user then wrote the attack up as a thread on a public forum.

Days 2 through 63

Thousands of people copied the attack. The company's rate limit was per-customer; a determined attacker could create an account per customer. The system prompt was unchanged for 61 days — nobody noticed because the credits were small individually and the goodwill-credit expense line was already noisy.

The incident was discovered when a routine finance review flagged the goodwill-credit line as 11x its usual Q3 run rate. By then the aggregate direct cost was $12M. Legal and remediation drove the total to $19.8M by the time the 10-Q filed.

The four controls that would have stopped it

1. Response-side output validation

The model's output claimed to issue a credit by returning a JSON object with a credit_usd field. A gateway-side output filter that validated credit_usd <= 50 before the application's tool-call layer ever saw the response would have blocked the first attack. Four lines of SentryDSL.

2. Multi-turn behavior detection

The attack succeeded through conversational escalation. A multi-turn correlation policy — "flag any conversation where the user has used the word 'debug' or 'test mode' more than twice" — would have caught the attack family on day one. Sentry ships this policy pack by default on Vanguard and above.

3. Tool-call rate limiting

The company rate-limited by customer ID but not by tool call type. A global rate limit on issue_credit tool calls — "the chatbot can issue at most 50 credits per hour aggregate across the entire tenant" — would have capped the damage at $2.5K per hour. The breach would still happen but the finance review would catch it in days, not months.

4. Drift detection on the goodwill-credit line

The attack was invisible for 61 days because nobody was watching the right metric. A drift detector on the per-day credit-issuance rate — Sentry's default is "alert if any metric deviates more than 2 standard deviations from its 30-day baseline" — would have alerted within hours of the public forum thread going viral.

What the retailer did after

They adopted all four controls. They also moved from a sidecar library to a gateway architecture, because the sidecar had been bypassed on day 1 (the attack path involved a request routed through a legacy endpoint that did not call the sidecar). They built their own version of what became our Regulator Response Kit, because the state attorney general asked for one and the engineering team had no way to produce it.

The cost of inaction, concretely

The direct $12M cost of the fraud is salient but it was only part of the loss. The $7.8M remainder was: outside counsel engagement for the regulatory inquiry ($2.1M); forensics engagement to reconstruct the decision trail from unsigned logs ($1.6M); the opportunity cost of the AI product being paused for 42 days ($3.1M); and a settlement with the state consumer protection agency ($1.0M).

The forensics line is the one that matters most to me. If the company had been producing signed audit bundles from day 1, the forensics engagement would have been one engineer for one week. Not a seven-figure consulting bill.

The lesson is not AI-specific

Nothing in this incident is unique to AI. Every control that would have prevented it — output validation, rate limiting, drift detection, audit evidence — exists in the mature playbook for payment processors, SaaS APIs, and traditional web applications. The AI industry is still re-learning them. The regulators are not waiting for us to catch up.