I was a part-time DBA – until a fabulous failover foul-up • The Register
Who, Me? No two mistakes are the same, but The Register thinks they’re all worth celebrating each Monday when we serve up a fresh edition of Who, Me? – the reader-contributed column in which we share your most magnificent messes, and your means of making it out alive.
This week, meet a reader we’ll Regomize as “Derek” who told us that around 2012 he worked as a part-time database administrator for a company that operated two MySQL instances – one in Boston, USA, and the other on the charming Mediterranean island of Malta – and set up symmetrical replication between them.
Derek thought this was a bad idea because Malta is not regarded as a global connectivity hub, and he felt the goal of symmetrical replication – swift mirroring of any changes made in either location – might be hard to achieve if something damaged a submarine cable to the island.
He was right for the wrong reasons, and learned that when the company’s Boston datacenter tested its automated systems for switching to backup power in case of emergency.
That test failed, leaving the Maltese database handling mission-critical, high-volume transaction applications.
Once the Boston bit barn came back online, Derek knew he would need to use the REPAIR TABLE command to catch up on changes made in Malta and restore synchronization between the two databases.
He did so but forgot the LOCAL option that would ensure he only repaired the Boston database. To be fair, the company’s manual only mentioned this on page two.
Remember that symmetrical replication? Derek had just started repairs to both the Boston and Malta databases, and the latter didn’t need to be repaired. It needed to run, unimpeded, to keep the business humming.
After a frantic hour of repairs and backtracking, Derek restored operations.
“The moral of this story is read the second page of the manual,” he told Who, Me?
This story has a happy ending of sorts because after Derek’s mistake, his employer recognized it probably needed a full-time DBA.
Yes, Derek should have RTFM. Has failing to do so led you into trouble? Here’s a clear instruction you can’t miss: if you have a story in this vein click here to send email to Who, Me? We could use a few more stories right now so get clicking! ®
