The Servers Are Burning
An engineer pushes a small change and OkCupid goes up in flames.
Hasnain says:
"How could such a tiny change have such an outsized impact on the site? “That same story happened so many different times,” my old boss David told me. “Someone launched a small, relatively innocuous change that did one of the millions of unexpected things it could have done, which then happened to break some part of the site, and then bring it all down—sometimes bring it down to the point where we couldn’t recover it for hours.”"
Posted on 2018-09-07T05:53:10+0000