Signed evidence 101: what a regulator actually wants

The questions auditors ask about AI systems are not technical, but the answers always are. Here is the single most common question in an AI governance audit, and the shape of the artifact that answers it.

CALLOUT

"Can you prove this specific decision, on this specific day, was consistent with the policies you had in effect at that moment?"

If you can produce one artifact that contains all of that, cryptographically bound together, the auditor moves on. If you cannot, you spend four engineering weeks reconstructing it after the fact, and the finding is already written.

The anatomy of a signed evidence bundle

A Sentry bundle contains six things, in this exact order, concatenated into a single tar.gz and signed with HMAC-SHA256 over the whole archive.

• The decisions file — line-delimited JSON, one row per decision, with the prompt, redacted prompt, response, policy chain, latency, model, provider, user reference, and timestamp.
• The policy bundle manifest — a JSON index of every policy in effect during the period, each listed with its SHA-256 content hash.
• The policy source files — the raw SentryDSL for each policy, so an auditor can read what the policy said in plain English.
• The compiled policy bytecode — the exact byte sequence the Gateway was executing. (This is what matters legally; the source is for human review.)
• A metadata file — tenant, period start/end, generator version, build commit SHA, and the KMS key alias used for signing.
• The signature file — the HMAC output over the SHA-256 digest of the archive, signed with a KMS-held key.

Why hashing matters

The policy manifest contains a SHA-256 for each policy. Each decision in the decisions file references those SHAs. This means an auditor can pick any decision at random, look up its policy references, hash the policy source they see in the bundle, and verify the hashes match. That proves the decision was produced by the policy you say it was produced by — because the hash can't be forged post-hoc.

If you log a decision ID but not the policy hash, every one of those decisions is an open invitation to the auditor to ask "how do I know you weren't running a different policy at that moment?"

Why signing matters

The HMAC signature covers the entire archive, including the manifest. If anyone — your team included — modifies any file in the bundle after the fact, the signature breaks. The auditor's verification tool reports the break. The artifact is now inadmissible.

This is the point. The signed bundle is designed to be hostile to tampering by the organization that produced it. That property is what makes a regulator trust it.

Common pitfalls

• Logging prompts and policy IDs, but not policy hashes. Prompts without a verifiable policy version are a log, not evidence.
• Keeping the signing key in the same system that produces the bundle. The key must live in a separate KMS so a Sentry compromise can't backdate bundles.
• Retention gaps. If your retention policy evicts a bundle before the regulator asks for it, you fail the lookback. EU AI Act high-risk requires 10 years.
• No chain-of-custody from decision to bundle. If a decision log can be edited independently of the bundle it rolls up into, the bundle's integrity is theoretical.

How Sentry handles the pitfalls

Decisions are written to an append-only store at Gateway side — there is no update path exposed to anyone, including Sentry operators. Bundles are produced on a schedule (daily by default) and the signing key lives in your KMS, never in a Sentry pod. Retention is enforced at the S3 Object Lock layer, so a rogue admin deleting bundles trips an alert and leaves a forensic trail.

The one thing to take away

When the regulator asks their one question, the answer is a URL to one tar.gz. Everything else — the forensics, the explanations, the chain-of-custody — is a property of that single file. If your audit answer is more than one file long, you are doing it the hard way.