A policy change gets approved at 4pm. It needs to reach 600 OPA instances spread across four clusters in three regions.
Someone asks three questions. How long until every instance is enforcing the new rule? How do you know none of them silently kept the old one? And if an auditor asks what the system decided at 4:15pm, where exactly do you look?
None of these are Rego questions. You can write a flawless policy and still fail all three. They are questions about distribution, verification, and observability, which means they are questions about architecture.
This is the part of OPA that most teams discover late, usually during an incident. The policy engine itself is the easy half. The hard half is the machinery that gets policy to the engine, proves it arrived intact, and records what the engine did with it.
The big picture: an agent, a contract, and a control plane
OPA's central architectural decision is that it ships an agent and a contract, but leaves the server side of that contract open.
The agent is the `opa` binary. It evaluates policy, and it speaks four management APIs: bundles for receiving policy and data, decision logs for reporting what it decided, status for reporting its own health, and discovery for receiving its own configuration.
Those four APIs are the contract. They are ordinary HTTP, deliberately simple, and OPA does not care what implements the other end. It can be an S3 bucket, an nginx container serving static tarballs, a bespoke Go service, a commercial product, or the OPA project's own control plane implementation. The agent behaves identically in every case.
This is why the OPA ecosystem looks the way it does. Large organizations built their own bundle pipelines for years because the contract was public and the server side was not prescribed. That freedom was genuine, and so was the cost: everyone rebuilt the same bundle server.
That gap has since been filled. The OPA project now maintains OPA Control Plane (OCP), a subproject that builds bundles from Git repositories, pulls external data in at build time, distributes to object storage such as S3 or Google Cloud Storage, and injects organization-wide policies into bundles based on label selectors. It is still young and worth evaluating on its current release rather than its promise, but it means the reference answer to "what serves the bundles" is no longer "write it yourself."
The architecture below holds regardless of which control plane you pick, because the contract is the stable part.
Inside a single OPA instance
The agent is a single Go process with four internal concerns: an API surface, a store, a compiler, and an evaluator. Around those sit the management plugins.
The store is worth dwelling on, because it constrains everything else. It is entirely in memory. There is no disk-backed index, no query planner reaching out to a database, and no cache tier. Base data arrives as JSON and policy arrives as a compiled abstract syntax tree, and both sit in process memory for the lifetime of the bundle.
Bundle activation happens inside a store transaction. Until that transaction commits, the previously activated policy remains live and continues serving requests. This is what makes hot policy reload safe: there is no window where a request sees half of the old bundle and half of the new one.
The plugins run on their own timers, entirely off the request path. The bundle plugin polls and writes into the store. The decision log plugin drains a buffer and uploads. The status plugin reports. A request being evaluated never blocks on any of them.
Bundles: the distribution layer
A bundle is a gzipped tarball containing `.rego` files, `data.json` files arranged into a directory hierarchy that mirrors the `data` document, and an optional `.manifest`.
{
"revision": "git-8f3a1c9",
"roots": ["authz", "billing/limits"]
}
The `revision` is opaque to OPA. It is a label you choose, and its only job is to appear in status reports and decision logs so you can correlate a decision back to the exact policy that produced it. Setting it to a Git SHA is the single highest-value line of configuration in this whole architecture.
The `roots` field is a claim of ownership. It declares which subtrees of the `data` document this bundle governs. Anything under a declared root that is not in the bundle gets erased on activation, which is how deletions propagate.
Roots also make multiple bundles possible. A platform team can own the `kubernetes` root while an application team owns `authz`, each shipping independently to the same OPA instance. The critical constraint is that roots must not overlap. If two bundles claim the same root, OPA enters an error state, and because there is no ordering guarantee about which bundle loads first, the failure is nondeterministic across your fleet.
Distribution is pull-based and polls on a configured interval.
services:
control-plane:
url: https://policy.internal.example.com/v1
bundles:
authz:
service: control-plane
resource: /bundles/authz.tar.gz
polling:
min_delay_seconds: 60
max_delay_seconds: 120
signing:
keyid: prod-bundle-key
status:
service: control-plane
decision_logs:
service: control-plane
reporting:
min_delay_seconds: 30
max_delay_seconds: 60
The polling interval is expressed as a range rather than a fixed number, and OPA jitters within it. With 600 instances on a fixed 60 second timer you would get a thundering herd every minute. The range spreads that load out.
Each poll sends the last activated revision in an `If-None-Match` header. An unchanged bundle returns `304 Not Modified` and costs a few hundred bytes, so a one minute interval across a large fleet is cheap.
There is a second bundle type. Delta bundles carry a `patch.json` containing JSON Patch operations applied to the in-memory store rather than a full snapshot. They exist for data that changes often enough that reshipping a large snapshot is wasteful.
Delta bundles come with sharp constraints. They can update data only, never policy. They cannot be signed. And they are not persisted to disk even when disk persistence is enabled for the bundle, which means a restart loses them and the instance falls back to the last full snapshot. Use them for volatile data, not as your primary distribution mechanism.
Signing and the chain of trust
A bundle server is, by construction, a thing that can inject arbitrary authorization logic into every service you run. Compromising it is equivalent to compromising all of them at once.
Bundle signing addresses this. A signed bundle contains a `.signatures.json` file holding a JWT whose payload enumerates every file in the bundle with its SHA-256 hash.
{
"files": [
{
"name": "authz/policy.rego",
"hash": "0d4dadd501f604412e82d9ab20e0bb6038fc07c1925d058686411ca5e4fea138",
"algorithm": "SHA-256"
},
{
"name": "authz/data.json",
"hash": "5ae75f6e54250e2060e65166845a69152436fd59c6873df7ff29c53813598cfa",
"algorithm": "SHA-256"
}
],
"scope": "prod"
}
Verification checks the JWT signature against a public key OPA holds out of band, then checks that the file list matches the bundle exactly and every hash agrees. The match is bidirectional, so an attacker can neither modify a listed file nor smuggle in an unlisted one. If verification fails, OPA refuses to activate, keeps serving the previous revision, and reports the failure through the status API.
The `scope` claim is easy to overlook and worth setting. It must match a value configured on the agent, which lets you sign staging and production bundles with the same key while making a staging bundle unusable in production.
Two limitations deserve attention. First, signature verification only happens when OPA is running in bundle mode or downloading bundles. Development commands including `opa eval` and `opa test` do not verify signatures, so a signed bundle is not a guarantee about what a developer ran locally. Second, delta bundles cannot be signed at all, so enabling them creates an unsigned path into the store.
Discovery: configuring the configuration
The configuration above is baked into each agent at startup. Changing a polling interval or adding a bundle means redeploying every instance, which reintroduces exactly the problem bundles were meant to solve, one level up.
Discovery fixes this. The agent boots with a minimal bootstrap config pointing at a discovery bundle. That bundle contains the rest of the configuration, and it can itself be a Rego policy evaluated against the agent's labels, so one discovery bundle can hand different configuration to different instances based on region, environment, or team.
Two rules govern it. The `discovery` section itself is immutable through discovery, which prevents an instance from being redirected to a different control plane by the control plane it currently trusts. And label changes are additive only: discovered labels are merged onto the bootstrap labels rather than replacing them, so an instance cannot disguise its own identity in status reports.
The security consequence is the important part. The discovery bundle supplies the verification keys used to check regular bundles. Anyone who can modify an unsigned discovery bundle can substitute their own key and then ship any policy they like, fully verified. Sign the discovery bundle, or the rest of the signing chain is decorative.
The evaluation path
Policy evaluation looks expensive if you imagine it as interpreting a rule set on every request. OPA's performance comes from doing as much as possible once, at activation, rather than repeatedly.
The first mechanism is rule indexing. At compile time, OPA analyzes equality expressions across a rule set and builds a trie. At request time it walks that trie with the input and retrieves only the rules that could possibly apply.
The effect is that a policy with a thousand `allow` rules does not evaluate a thousand rules. If the input path and method select three of them, three are evaluated. Adding rules that do not match a given input costs that input nothing.
Indexing only works on rules written in an indexable form, which is why "write policies with indexed statements" is standard OPA performance advice. A rule that opens with a comparison against a concrete field indexes well. One that opens by iterating a large collection does not, and the indexer falls back to evaluating it every time.
The second mechanism is early exit. When a rule set can only produce one ground value, as with a set of boolean `allow` rules, OPA stops as soon as one match is found. Further iteration cannot change the outcome, so it is canceled.
The third and most powerful is partial evaluation. You mark certain inputs as unknown, and OPA evaluates everything that does not depend on them, emitting a residual policy that only contains the parts that genuinely needed request-time information.
Running this at build time with `opa build --optimize` collapses data-driven policy into rules the indexer handles well. A role-based policy that scans a large role table on every request becomes a set of directly indexed rules. Evaluation latency then stays roughly flat as the underlying data set grows, which is the outcome you actually want.
Partial evaluation is not free, and the cost lands in the right place. Both the time and the memory it consumes grow with the data set, but they are spent during the build, outside the request path. The only thing that gets slower is how quickly a policy change propagates.
There is a second use for partial evaluation that has nothing to do with speed. If you mark the rows of a database as unknown, the residual policy that comes back is effectively a filter condition, which can be translated into a SQL `WHERE` clause. This is how OPA is used for data filtering: rather than asking "may this user see this row" a million times, you ask once and get back the predicate that describes every row they may see.
Decision logs: the audit trail
Every decision OPA makes can be emitted as an event carrying the queried path, the full `input`, the result, the bundle revision that produced it, a `decision_id`, and the agent's labels.
That combination is what makes decisions reproducible. Given an event, you have the exact policy version and the exact facts, so you can replay it. Replaying historical traffic against a candidate policy before shipping it is the closest thing this architecture has to a staging environment.
The obvious problem is that `input` frequently contains things you must not centralize. OPA handles this with a masking policy, written in Rego and evaluated against each event before upload.
package system.log
mask contains "/input/password"
mask contains "/input/headers/authorization"
mask contains {"op": "upsert", "path": "/input/user/ssn", "value": "REDACTED"}
Plain pointers erase the value. The object form replaces it, which preserves the shape of the event for downstream consumers that expect the field to exist. Erased paths are recorded in an `erased` array on the event itself, so the audit trail states honestly that redaction occurred.
A companion `drop_decision` policy discards entire events before they leave the process, which is how you suppress high-volume health check decisions without losing the ones that matter.
Upload is buffered and batched. OPA adapts its chunk size against a configured hard limit, scaling up while it has headroom and re-encoding when it overshoots. Two behaviors under pressure are worth knowing. If the buffer fills, events are dropped, and the count of dropped events is reported through the status API. If a single event is too large to fit in a chunk, OPA discards the non-deterministic builtin cache from it and retries.
Both are deliberate: the decision log plugin degrades rather than blocking the request path. That is the correct trade for authorization, but it means decision logs are not a guaranteed-delivery audit channel by default. If you need every event, you need to monitor the dropped-event counter and provision the buffer accordingly.
A request, end to end
With the components in place, here is a single authorization decision from the caller's request through to the audit record.
The path from request to decision touches no network beyond loopback, no disk, and no database. That is the whole reason this architecture can sit in front of every request rather than beside occasional ones.
The `decision_id` returned alongside the result is the thread that ties it together. Log it in your application, and an investigation months later becomes a lookup rather than an archaeology project.
Failure modes and fault tolerance
Most OPA failures are not evaluation failures. They are distribution failures, and the dangerous ones are silent.
The empty policy trap. An OPA that has loaded no policy does not error. It returns `undefined`, which over HTTP appears as an empty JSON object. A service that treats a missing `allow` field as false fails closed and merely breaks. A service that checks `if response.deny` fails open and authorizes everything. This is the single most consequential failure mode in the architecture, and the fix is at the boundary: gate readiness on `/health?bundles=true` so an instance never receives traffic before its bundles have activated.
Bundle staleness. If the bundle server becomes unreachable, OPA keeps serving the last good revision indefinitely. This is correct behavior and exactly what you want during a control plane outage, but it means a broken pipeline is invisible from the data plane. Everything keeps working, on policy that is quietly getting older. The status API is the only place this surfaces.
{
"bundles": {
"authz": {
"name": "authz",
"active_revision": "git-8f3a1c9",
"last_successful_download": "2026-03-14T09:41:02Z",
"last_successful_activation": "2026-03-14T09:41:02Z",
"last_request": "2026-03-14T14:07:55Z"
}
}
}
Alert on the gap between `last_request` and `last_successful_activation`. When downloads keep succeeding but activations stop, you are looking at a bundle that compiles on someone's laptop and fails in production. Alerting on staleness is not optional in this architecture; it is the only signal that policy distribution has stopped.
Root conflicts. Two bundles claiming the same root put OPA into an error state, and because load order is not guaranteed, some instances may land in that state while others do not. A fleet where 40 percent of instances are wedged and the rest are fine is a confusing thing to debug at 2am. Treat root allocation as a design decision with a single owner.
Memory ceilings. The store is in memory and scales with both rules and data. As a rough calibration, an ACL-style policy with 10,000 rules sits around 130MB, while the same policy at 100,000 rules approaches 1.1GB. Multiply by every sidecar in the fleet. This is the constraint that decides whether a data set belongs in OPA at all, and partial evaluation makes it worse before it makes it better, because it trades additional generated rules for faster evaluation.
Signature verification gaps. Signing protects the bundle download path only. Local development commands skip verification, and delta bundles cannot be signed. If your threat model includes a compromised control plane, enumerate every path into the store, not just the signed one.
Log loss under pressure. Decision logs drop when buffers fill. The dropped count is reported through status, which means it is discoverable but only if you look. A compliance requirement that assumes complete logs needs that counter wired to an alert.
The pattern running through all of these is worth naming. OPA's failure design consistently favors availability of the data plane over freshness and completeness in the control plane. Stale policy still serves. Dropped logs still serve. A failed signature still serves, on the old bundle. Every one of those is the right call for something sitting in the request path, and every one of them is invisible unless you monitor the control plane separately.
The complete picture
Assembled, the architecture spans four layers, each with a different change frequency and a different owner.
Summary
The thing to take away is that OPA solved the decision problem by turning policy into a pure function, and then discovered that the hard part was everywhere else.
Evaluation is genuinely fast, and the reasons are structural rather than clever: an in-memory store, a compile-time rule index, early exit, and partial evaluation that moves work off the request path entirely. None of that is where teams get into trouble.
Trouble lives in the four management APIs. Bundles decide how fast policy propagates and what happens when it cannot. Signing decides whether the propagation channel is a control point or an attack surface, and the discovery bundle is the root of that trust whether or not you signed it. Decision logs decide whether you can answer questions after the fact. Status decides whether you find out that any of the above stopped working.
The architecture is deliberately biased toward keeping the data plane serving. Stale policy still serves, dropped logs still serve, a rejected bundle still serves the previous revision. That bias is correct for something in the request path of every call, and it has one consequence worth internalizing: the data plane will almost never tell you that the control plane is broken. Monitoring the pipeline is not an operational afterthought here. It is the part of the design that OPA leaves to you.
Related on this blog: Architecture series