I spent the whole day dealing with servers. The facility where Wopular and MoviesWithButter are hosted were having some network issues. I was able to get MWB up and running within an hour or so. Wopular was a whole different story. The Database is about 21GB. It's a huge db, so whenever there's a network issue, I can always count on at least one corrupt table. And it always happens to be the largest table, which takes several hours to diagnose and repair.
I've had many outages in the past, so I kinda know the drill. This time was a little different though. The site couldn't write to one of the tables in the database. When it tries to, the whole site goes down.
I thought it had to be a corrupt table. Just my luck though, when it was done checking, my shell program crashed. I couldn't tell if the table was corrupt. I didn't want to spent another two hours rechecking the table, so I went ahead with the repair. That meant I had to clear up space on the server to make room for it.
When MySQL repairs a table, it creates a temporary table to store the results. This means if it's repairing a table that's 1GB, you'll need at least another GB of free space for that temp table. This is very important. If you run of space during the repair job, it could wipe out the entire table being repaired. I've lost a table in the past this way.
Anyway, while the server was busy deleting unwanted files to make room for the repair job, I head out for a little bit. When I got back with beer in hand and fired up the laptop, everything was ... surprisingly working. I didn't even have to do the repair.
Odd. But good. Maybe MySQL was doing the repair during this entire time, and that's why whenever I tried writing to the table, it won't let me. It's probably why my disk space was running low.
I don't know. I'm just glad everything's working again.
UPDATE 10/4/2011: Yesterday, the data center where this website is located was getting DDOS-ed and was out for hours throughout the day. That's the last straw for me. I'm moving this site to another data center.
