In all seriousness, I do appreciate what you are doing. Your stack and client code must be incredibly resilient if you can do this every week without much functional impact to the app. My reaction probably means that we’re far from this.

Did you get a chance to use the Simians? Your tests seem simple to implement, yet effective, and also give a head start in debugging.

Hi Sunny,

Isn’t it obvious? Failure and Friday start with an F. 🙂

More seriously, Fridays tend to naturally have fewer deploys, so we aren’t blocking the company doing testing on that day. Also, the failures we’re inducing are failures the system should handle, and once we remove that failure, we’re back to a working state.

That’s really cool.. in addition to starting to use PagerDuty ourselves we also recently switched to Percona’s XtraDB Cluster at BrightTag as well. Great minds must still think alike! Nice writeup, Doug – you’ve got us thinking about incorporating Failure Friday as well here.

Hi Emre,

I respectfully disagree. MySQL certainly has its share of silly shortcomings but those are known issues that have common workarounds. Both Google and Facebook use MySQL in their infrastructure and both are highly reliable.

To improve our own reliability of MySQL, we’ve recently switched to Percona’s XtraDB Cluster software. We now have a 3 node synchronous multi-master cluster for our application tier. In the near future, we will move to a multi-master model using that clustering software so cluster nodes can come and go with no impact to the application.

Yes. We have our alerting split into two groups. Most issues are delivered to us by PagerDuty. Critical issues (like the system being down or unable to deliver events) use an external monitoring system which phones everyone directly.

Hi Ryan, the primary reasoning was that we did not want a scheduled job to interfere with the attacks, e.g. having a Chef run override our firewall rules when we are doing the network isolation or having a Cassandra repair run in the middle of taking the service down.

For cron job failures, we have pretty limited monitoring in place right now, but we are actively looking to implement something like Proby for our purposes.