Larry Halff’s social media website ma.gnolia.com is gone.
He spent four years during which he, “devoted most of my time, energy, and love” to building the site. Users of the site had had built intellectual and social capital through the bookmarks, groups, and connections they made. Many users had even purchased premium feature subscriptions.
So from what we have learned, here is the autopsy:
Magnolia had about 500GB of data in a MySQL database. There were two instances of the database on two separate servers. Each server had a RAID configured to protect the data.
The primary database was replicated from the primary server to the secondary server using a firewire connection. Sounds good…so what went wrong?
The MySQL database suffered some file system corruption. We don’t know if that corruption was at the operating system level or within the MySQL database itself. Both scenarios are possible and lead to very similar outcomes. Assuming that reliable hardware was in use, there is not much that Magnolia could have done to avoid this problem.
But there were three significant mistakes which led to the demise of Magnolia.
First, whenever data is replicated, it MUST also be backed up. The distinction being that replication happens in near real-time, so any corruption in the primary data is replicated to the secondary data almost immediately. But most good backup solutions offer versioning, meaning you can drop back to an earlier version of the data – hopefully earlier than when the corruption occurred.
Second, Magnolia admittedly did not test their backup strategy. Given the scenario used, it would have been difficult to test, but it is more than prudent to test. You really must test.
Finally, Magnolia did not have any offsite data storage. Having a remote backup would have saved the day.
The following video tells the sad story:
Citizen Garden Episode 11: Whither Ma.gnolia? from Larry Halff on Vimeo.
We wish Larry and Magnolia’s users the best going forward.
on Mar 13th, 2009 at 6:06 pm
I’ve been an active Magnolia user for many years now. It’s a shame that they didn’t do some sort of backup of the data on a regular basis.
on Mar 20th, 2009 at 2:17 pm
Thanks for sharing Larry.
I’m an IT guy, and my philosophy has always been ‘If proper DR is in place and tested, you will never need it’.
The big question I ask is “if you came in today and your server room was burnt to the ground, what would you do?” This helps me and others come into the proper DR thinking mode and put needed procedures in place. Get your vendors, phone numbers, equipment numbers/ID’s, all in one place and take a shapshot of your server configuration.
It can be a bit of work at first, but once in place, needs only minor maintenance and helps me sleep better at night.
When I was a young IT lad, I completely stepped on (overwrote) an entire database costing about 1,000 man hours. I had to explain to the team my error and why it happened. I’ll never do that again thanks to proper DR and more mindful actions.
Cheers -Tim
on Mar 23rd, 2009 at 10:52 pm
I’m resolved that I cannot get my data restored. I’m thankful I got back my public bookmarks in HTML format through the recovery tools. What I’m a bit frustrated by is Larry’s silence since that interview.
Is he going to reopen Ma.gnolia at any time? I really did appreciate the interface and features of it and would love to start rebuilding once again. But the question remains, is he going to re-open shop? Because if he isn’t, I’d like to know so I can take my bookmarks to some place like Delicious (but I’d rather not).
on May 19th, 2010 at 8:53 pm
[...] you must be a member to see what people have bookmarked and to save your own. I found the”Magnolia Autopsy” article on why the site [...]