It's 2 a.m. Your on-call phone goes off. A payment service is down, and nobody can explain why. It worked fine in staging. It passed every test. But somewhere in the tangle of microservices, a database connection pool quietly exhausted itself, and the fallback that was supposed to kick in never did. Nobody knew the fallback was broken. It had never been tested in production. Not once.
This is the failure mode that chaos engineering was built to catch: not the bugs you write tests for, but the assumptions you never thought to question.
The problem with waiting for things to break
Traditional reliability engineering is defensive. You write tests, you build redundancy, you add monitoring. Then you hope. The mental model is: failures are rare, bad, and to be avoided. If you build carefully enough, the system won't break.
The problem is that distributed systems don't behave that way. In a system with dozens of services, databases, caches, queues, and third-party APIs, failure isn't rare. It's constant. Individual components fail all the time. The question isn't whether something will fail; it's whether the rest of the system will handle it gracefully when it does.
Staging environments can't answer that question reliably. They don't carry production traffic patterns. They don't have the same data volumes. They don't reproduce the subtle timing issues that emerge under real load. A system can look perfectly healthy in staging and collapse in production under conditions nobody anticipated.
Netflix learned this the hard way in 2008, when a database corruption event took down their DVD shipping operation for three days. The incident forced a rethink. Rather than just adding more safeguards and hoping, they asked a different question: what if we stopped hoping and started testing?
Chaos Monkey: break it before it breaks you
In 2010, Netflix engineer Greg Orzell built Chaos Monkey: a tool with one job. It scanned Netflix's production infrastructure, picked a running server at random, and terminated it. During business hours. While real users were streaming.
The reasoning was deliberate. If servers are going to fail in the middle of the night, you want to know your system handles it correctly while your engineers are awake and can respond. Chaos Monkey turned that assumption into a daily test.
The results were uncomfortable. Services that were supposed to failover didn't. Retry logic that looked correct on paper had subtle bugs. Dependencies that engineers assumed were non-critical turned out to be load-bearing. Each failure Chaos Monkey triggered was a failure that would otherwise have happened at 3 a.m., silently, with no engineer watching.
Over time, the Netflix team noticed something else: engineers started writing services differently. Knowing that any instance could disappear at any moment, they stopped designing for stability and started designing for recovery. Timeouts, retries, circuit breakers, and graceful degradation became defaults rather than afterthoughts.
From tool to discipline
Chaos Monkey was a specific tool built for a specific problem. What it sparked was something broader: a set of principles for testing the resilience of distributed systems that the industry now calls chaos engineering.
The core idea is deceptively simple. You form a hypothesis about how your system should behave under a failure condition. You inject that failure in a controlled way. You observe what actually happens and compare it to what you expected. If the system behaves as expected, you've confirmed a resilience property. If it doesn't, you've found a weakness before your users did.
This makes chaos engineering fundamentally different from traditional testing. Unit tests verify that code does what the developer intended. Integration tests verify that services talk to each other correctly. Chaos engineering verifies that the system as a whole survives conditions the developer never anticipated. It is not a test of correctness. It is a test of resilience.
The Simian Army: expanding the blast radius
Chaos Monkey only killed instances. Netflix quickly found that instance failure was just one category of production problem. They built the Simian Army: a suite of tools each designed to simulate a different class of failure.
Latency Monkey injected artificial delays into service calls, simulating slow dependencies. This exposed services that had no timeouts, or timeouts set so high they were effectively useless. Chaos Kong went further still: it simulated the failure of an entire AWS availability zone, forcing traffic to reroute across regions. Conformity Monkey looked for instances that violated best-practice deployment rules. Janitor Monkey hunted unused resources and cleaned them up automatically.
Each tool was testing a different assumption. Chaos Monkey tested: "can we lose a server?" Latency Monkey tested: "can we survive a slow dependency?" Chaos Kong tested: "can we survive losing an entire data center?" The principle was the same across all of them: don't assume resilience. Prove it.
Blast radius and the safety controls that make this viable
The obvious objection to chaos engineering is that it sounds reckless. Deliberately breaking production systems while users are on them seems like a recipe for outages, not a cure for them.
The answer is blast radius control. A chaos experiment is not a random act of destruction. It is a scoped, time-limited, observable intervention. You define in advance exactly which services are in scope, what the failure will be, when it will run, and what your rollback procedure is. You run experiments during business hours, when engineers are watching. You start small: one instance in one region, for a short window. You abort immediately if a defined safety threshold is breached.
The key precondition is having steady-state metrics: a clear baseline of what "healthy" looks like. Before you run an experiment, you need to know what normal request latency, error rate, and throughput look like. Without that baseline, you cannot tell whether the experiment caused a degradation or just exposed one that already existed.
This is also why chaos engineering is not appropriate for every team. If your observability is weak, if you don't have clear ownership of services, or if your incident response process is immature, you're not ready to deliberately inject failures. The discipline requires a foundation of monitoring and operational maturity to be safe and useful.
Where chaos engineering lives today
Netflix's ideas have spread far beyond Netflix. Chaos engineering is now a standard practice at large engineering organizations, and a growing expectation at any company running critical distributed systems. The tooling has matured considerably since the Simian Army.
Gremlin is the most widely adopted commercial platform. It provides a catalog of pre-built failure scenarios, blast-radius controls, and integration with major cloud providers, making chaos engineering accessible to teams that don't want to build their own tooling. Chaos Toolkit is an open-source alternative that takes a declarative approach: you describe your experiment as a JSON or YAML file, and the toolkit executes and evaluates it. LitmusChaos is purpose-built for Kubernetes environments, integrating directly with the cluster to inject pod, node, and network failures at the container orchestration layer.
The practice has also expanded in scope. Early chaos engineering focused on infrastructure: kill a server, drop a network link. Modern chaos engineering increasingly targets the application layer: simulate a slow third-party API, exhaust a connection pool, inject corrupt data into a message queue. The closer the failure is to the kinds of problems that actually happen in production, the more useful the experiment.
Regulatory environments have taken notice too. Financial services firms in several jurisdictions are now required to demonstrate resilience through testing. Chaos engineering, under the label of resilience testing or fault injection testing, has become part of how organizations satisfy those requirements.
Summary
Chaos engineering starts from an honest premise: in a distributed system, failure is not exceptional. It is normal. The question is whether you discover your failure modes in a controlled experiment with engineers watching, or in a production outage with customers affected.
Chaos Monkey was the first serious answer to that question. The Simian Army expanded it into a full discipline. That discipline is now formalized, tooled, and practiced at scale across the industry.
The insight that made it work was not technical. It was cultural: a willingness to treat production as a place where you learn, not just a place where you hope. You define a hypothesis, inject a failure, observe what happens, and fix what breaks. Then you do it again. Each cycle either confirms that your system is resilient, or finds a weakness before your users do. Either outcome is a good one.
Part of the Explained series — concepts in tech, clearly.