ezBoard experienced a hack into their systems last week that decimated all their hosted forums. It wasn't the first attack on message boards in the last year: over 30,000 phpBB boards were erased by a virus that took advantage of a flaw in that software. Most people forget to update their software, so while Google ended the attack (they were used by the virus to search for other hackable sites) many are still vulnerable.
I feel for the ezBaord crew. It really hurts when your service is not online as expected all the time. For Chris and I, we have been doing this with Network54 for so long that when anything goes wrong, it is an intensely emotional experience.
News of the whole event made us re-evaluate our backup strategy which resulted in a couple of small but important changes. One of which is to store some backups outside of California. It's earthquake country here after all!
Luckily, we don't store messages in the file systems of hundreds of servers. At first glance it might seem like a good idea since one system failure would only take out a small percentage of data. But if you run a grid like operation like we do (and I assume they do), then there is the idea of "Catastrophic Failure" where a virus or hacker on one machine can use the grid tools to spread destruction to all the others. Not only that, but I have no idea how to backup data in such a way. It would require skills we don't have.
So we use a database for all our data and messages (minus images, something we'll fix in the future). Databases allow us to stream changes to other machines running databases such that the other servers continuously reconstruct complete and up-to-the-second backups. We have a few of them in different locations. One is right on site in case the main database cluster goes down (in this situation it takes over immediately and you get a read only version of the site while we get notified). The others take snapshots at various intervals and keep the logs that update from one snapshot to the other -- this way we can go back to any time between snapshots and recover. And recovery is fast (instantaneous if the on site backup was OK, longer if we have to drive to the datacenter -- this is Los Angeles so driving time is an important consideration).
We never swimmed in millions of investor dollars, and we know that our data is your data and without you we would be a ghostown of the internet, so we have and will put our paranoia into data protecting action. Can't afford not to.
N54/Steven Roussey/
My Weblog