Gateway vs sidecar: the architectural choice that decides your AI audit

A friend asked me last week why Sentry runs as a gateway, not as a sidecar in each application pod. The question is architecturally sharp. Here is the long answer, because the short one — "audit" — is correct but unsatisfying.

The two models

Policy enforcement for AI happens somewhere in the call path between your application and the model provider. There are two sensible places to put it.

• Gateway: a stateless proxy that terminates all AI calls at the network edge, enforces policy, and forwards to the provider. Your application points its OpenAI SDK at sentry.yourco.com instead of api.openai.com.
• Sidecar: a library (or sidecar container) that wraps the provider SDK inside your application process. Your application still calls OpenAI directly, but every call runs through the library's policy check first.

The sidecar wins on three dimensions

The sidecar architecture has real advantages. Latency: zero network hops, so you add microseconds not milliseconds. Developer ergonomics: a drop-in library is easier to adopt than a new network endpoint. Isolation: your governance code runs inside your trust boundary.

The gateway wins on the one that matters

The gateway wins on reproducibility. When an auditor asks "what policy ran on this specific request", the gateway can answer with cryptographic certainty: the exact bytecode version is tagged on the decision record, and the request literally cannot have reached the provider without passing through that version.

With a sidecar, the application process could in principle be running a stale library version, a monkeypatched version, or no library at all. Even if 99.9% of your pods are compliant, the auditor can produce a hypothetical where one is not — and you cannot prove the hypothetical is empty.

CALLOUT

The gateway's superpower is that it cannot be bypassed by accident. The sidecar's weakness is that it can — and "accident" is exactly what auditors assume.

The latency objection, debunked

The usual argument against the gateway is latency. In our production data the p95 added latency is 3.8ms. The slowest LLM call we have ever seen is 7 seconds. The gateway contributes 0.05% of total latency. Nobody has ever noticed.

If you are latency-bound in a way where 4ms matters, you are probably doing streaming inference, in which case the gateway terminates the request and streams back — the overhead is on connection setup, not throughput.

The hybrid we actually use

In practice, large customers run the gateway for every production call and a lightweight sidecar for in-process pre-screening in latency-critical paths. The sidecar does the fast patterns, the gateway does the authoritative decision. The signed audit record always comes from the gateway. That is the configuration that passes an audit and still feels fast.

The decision rubric

If you take only the rubric, here it is:

• If you need to produce signed, reproducible evidence to a regulator, use a gateway.
• If you need pre-screening in a latency-critical path, use a sidecar in front of the gateway.
• If you only want visibility and no enforcement, use a sidecar.
• If you only sometimes need enforcement, you need a gateway you can bypass selectively — this exists but is rare.

The real question

The sidecar vs gateway question is a stand-in for "am I building AI with meaningful oversight, or just logging it". The sidecar is often a comfort blanket for logging. The gateway is a commitment to enforcement. Which commitment you need is a product and compliance question, not a performance question.