Netflix’s Chaos Monkey 🐒
What is Chaos Engineering?
No, it’s not about the chaos that arises when the production goes down; rather, it is a powerful practice that can prevent the “downtime chaos”. Here’s the gist:
Remember “The Office” episode where Dwight tries to check fire safety preparedness of his co-workers by lighting an actual fire in a trash can, sealing the office exits shut and cutting the phone lines. Well, that “controlled fire” summarises Chaos Engineering.
Instead of waiting for your system to break under unexpected pressure, you proactively introduce controlled disruptions to see how it responds. It’s like giving your system a stress test but with more intention and planning.
The goal is to identify weaknesses and build safeguards before they cause real-world problems. Think of it as preventive medicine for your software!
Here are some key points about chaos engineering:
- It’s about controlled experiments: You don’t just randomly break things. You carefully plan and execute experiments that target specific areas of your system.
- It’s not about creating chaos: It’s about gaining valuable insights and building confidence in your system’s ability to handle the unexpected.
How to make your system Fault Tolerant? Crash it! — Enter Netflix’s Chaos Monkey
But… Where’s the Monkey?
The Netflix Chaos Monkey is a specific tool used within the practice of chaos engineering, created by Netflix to test the resiliency of their own systems. Here’s how it works:
Imagine a mischievous monkey wreaking havoc in your server room, physically terminating the systems and breaking cables. Simulating unexpected failures exposes engineers to the pain of losing servers and motivates them to build resilient services that can handle such disruptions.
Benefits of Chaos Monkey:
- Improved system resilience: By proactively identifying and fixing weaknesses, Chaos Monkey helps prevent outages and ensures smooth operation under unexpected situations.
- Faster recovery: It encourages building systems that can automatically detect and recover from failures, minimizing downtime.
- Increased confidence: Engineers gain confidence in their software’s ability to handle real-world challenges by continuously testing and validating system resilience.
The Simian Army
As a result of implementing Chaos Monkey, Netflix saw a dramatic change in their approach to system design. Engineers started planning for breakdowns for every single microservice they were working on.
Inspired by the success of the Chaos Monkey, it triggered the creation of new simians that induce various kinds of failures, detect abnormal conditions, and test the ability to survive them.
Takeaways
A blog by Netflix defines their fault-tolerance strategy as: ‘The best way to avoid failure is to fail constantly.’ Many larger tech companies practice Chaos Engineering to understand their distributed systems and microservice architectures better. The list includes some big names such as Twilio, Netflix, LinkedIn, Facebook, Google, Microsoft and Amazon.
If you’re interested in implementing chaos engineering in your project, several tools and approaches can help you achieve similar goals:
- Chaos Toolkit: https://chaostoolkit.org/reference/tutorial/
- Gremlin: https://www.gremlin.com/
- pChaos: https://github.com/tiagorlampert/CHAOS
To get into Chaos Engineering for a distributed system, an organization can follow these general steps:
- Enable Monitoring: Injecting chaos without a monitoring system totally removes the controlled context. It should already have a steady monitoring system to identify if there is a disruption in the system.
- Disable Single Nodes: You can start small by disabling a single node from the system. It can be your Payments Microservice or your recommendation system, and define how the complete system would behave.
- Go BIG!!: Next, create multiple permutations of node failures. What if both the Authentication and Payment Gateway fail?
- The Ultimate Chaos: The next step is completely disabling a Cloud Region. Although this is a blue moon event, Chaos Engineering is about expecting the unexpected.
- Chaos Injection: Once all the scenarios are well thought out, it’s time to randomize the chaos. Determine the frequency of deactivation of instances/clusters/databases and give it a go.
Remember, planning and executing your experiments carefully within a controlled environment is crucial. Start small, monitor closely, and prioritize learning and improvement based on your findings.
By embracing the principles of chaos engineering, you can build more resilient and reliable systems, even without using Netflix’s specific tools.
