The archive · Work & Ways of Doing · Technical decision · 2010–2012
Chaos Monkey: Netflix breaks its own service on purpose so streaming survives
Netflix built Chaos Monkey to randomly kill production instances so its cloud survives — then open-sourced it, seeding chaos engineering.
Netflix
What it had to solve
Netflix ran its streaming service on AWS, where instance failures are common and outages had bitten the industry hard; instead of waiting to be surprised, the engineering team decided to surprise itself.
How it works
Netflix ran its streaming service on AWS, and in the cloud, instances fail — hosts die, network blips, load balancers misbehave. Rather than wait for an outage to expose the weak spots, the engineering team built Chaos Monkey, software that randomly disables production instances during business hours, while engineers stood by to watch.
The name came from 'unleashing a wild monkey with a weapon in your data center to randomly shoot down instances and chew through cables — all the while we continue serving our customers without interruption.' The point was not the tool itself but the change in mindset: if your system must survive random daily attacks, you build automatic recovery instead of hoping nothing breaks.
In July 2012 Netflix open-sourced Chaos Monkey, and the idea spread. Coverage in TechCrunch and Wired made 'fail often' a known engineering motto; Stack Exchange's Jeff Atwood said his team only solved its outage problem after the 'Chaos Monkey chose them'. The practice was later formalized as chaos engineering — experimenting on production systems to build confidence they can withstand turbulent conditions.
Why it lands
- Random, unattended failures during work hours turn rare terrifying outages into routine drills with engineers ready.
- Failing often forces automatic recovery mechanisms, so a 3am failure stops being a crisis.
- Open-sourcing gave the idea legs — other companies adopted the practice, and it became a discipline.
- The name made the concept unforgettable, which is why the practice spread as fast as it did.
What it did
Chaos Monkey forced Netflix's architecture to tolerate instance loss without customer impact and turned failure drills into routine; the open-sourced tool got coverage in TechCrunch and Wired, Stack Exchange's Jeff Atwood credited the same idea for fixing his outages, and the practice grew into the discipline of chaos engineering.
What you can take
Don't wait for the outage you can't predict: induce small failures during the day, with engineers watching, so the system — and the team — learn resilience before a 3am disaster teaches them.
Since then
Netflix kept expanding the idea: Latency Monkey, Doctor Monkey, Chaos Kong for whole availability zones, and later the Chaos Automation Platform, with a dedicated chaos engineering team formed in 2015. The original Simian Army repo was retired in 2021, replaced by standalone tools, but the practice outgrew Netflix — chaos engineering became an industry discipline with books, conferences, commercial tools and adopters like Capital One.
Sources
- Netflix/SimianArmy — Tools for keeping your cloud operating in top form
- Netflix Open Sources Chaos Monkey — A Tool Designed To Cause Failure So You Can Make A Stronger Cloud
- Netflix Abuses Amazon With Monkeys. Now You Can Too
- What is Chaos Monkey? Chaos engineering explained
spotted an error? The archive wants to know.
Your turn
You just read one. Describe the brief you are staring at, and see who has been given the same problem.
Free account · 3 free questions · no card