Another Network Outage ... Argh ...

Thu, 09/29/2011 - 9:16pm — senh

I spent the whole day dealing with servers. The facility where Wopular and MoviesWithButter are hosted were having some network issues. I was able to get MWB up and running within an hour or so. Wopular was a whole different story. The Database is about 21GB. It's a huge db, so whenever there's a network issue, I can always count on at least one corrupt table. And it always happens to be the largest table, which takes several hours to diagnose and repair.

I've had many outages in the past, so I kinda know the drill. This time was a little different though. The site couldn't write to one of the tables in the database. When it tries to, the whole site goes down.

I thought it had to be a corrupt table. Just my luck though, when it was done checking, my shell program crashed. I couldn't tell if the table was corrupt. I didn't want to spent another two hours rechecking the table, so I went ahead with the repair. That meant I had to clear up space on the server to make room for it.

When MySQL repairs a table, it creates a temporary table to store the results. This means if it's repairing a table that's 1GB, you'll need at least another GB of free space for that temp table. This is very important. If you run of space during the repair job, it could wipe out the entire table being repaired. I've lost a table in the past this way.

Anyway, while the server was busy deleting unwanted files to make room for the repair job, I head out for a little bit. When I got back with beer in hand and fired up the laptop, everything was ... surprisingly working. I didn't even have to do the repair.

Odd. But good. Maybe MySQL was doing the repair during this entire time, and that's why whenever I tried writing to the table, it won't let me. It's probably why my disk space was running low.

I don't know. I'm just glad everything's working again.

UPDATE 10/4/2011: Yesterday, the data center where this website is located was getting DDOS-ed and was out for hours throughout the day. That's the last straw for me. I'm moving this site to another data center.

Sections:  note from the editor   
SENH'S LATEST BLOG ENTRIES
  • "Storm The Gates" Mobile Game
    My friends just launched a mobile game called "Storm the Gates"! It's available on both Android and iOS. I'm not much of a gamer, so I didn't know what to do when I started it. But my 9-year-old son turned it on and knew exactly what to do. More
  • "Ip Man 3" Has New Release Date in China and Posters to Prove It
    Over at MoviesWithButter.com, I’ve written two articles regarding “Ip Man 3,” the film that’s holding its own in Asian territories against “Star Wars: The Force Awakens.” If you’re not familiar with Asian cinema,”Ip Man” one of the most popular franchises in that area of the world. More
  • Live-Blogged The Golden Globes at MWB
    Yesterday, I live-blogged The Golden Globes -- for various reasons. First, it gets traffic. Second, I might as well do something useful while watching TV. Lastly, I was looking forward to seeing Ricky Gervais host the event again. Nothing against Tina Fey and Amy Poehler, but he’s my favorite. A google search revealed that the last time I did this for this award show was More
  • Kings vs. Mavs: Ominous Stats, News for Sacramento Before Game
    The preview on NBA.com and the injury report from SactownRoyalty.com don’t look good for the Sacramento Kings going into tonight’s game against the playoff-bound Dallas Mavericks. More
  • What?! Kings Beat Thunder, 116-104
    (Image from the Sacramento Bee. Click here to view more images from the game.) What a win. It was a wild one, especially in the first half. We [Sacramento Kings] were down by 17 in the first quarter. In frustration, I was about to turn off the TV, but the teaml called a timeout. More
More
SENH'S RELATED BLOG ENTRIES
  • Downtime & How to Make a Repair Job on MySQL Go Faster
    The site crashed yesterday morning at around 9am, although it’s hardly noticable until a couple hours later. I didn’t notice it until I started picking stories to feature on the homepage. Whenever i tried to feature an article, it would stop the site from loading up. From that, I knew that the MySQL database must be checking for corrupt tables. More
  • A Couple Week's of Down Time
    You may have noticed that the site has been pretty unstable lately. That's because I moved my site to another internet service provider. More
  • Moved SimplePie File Caching to Database
    A while back, My ISP told me my backups took half a day to complete, which is too long. If anything goes wrong during that period, the process would have to start all over again, which happened quite often. The customer rep told me that my server had too many files, and even though they didn’t take up much disk space when added up, the process of copying each one of them takes a lot of time. More
  • Relieving High Server Load by Blocking Search Bots
    Over the years, whenever the site was slow to load up, I always went through the same routine. I checked traffic logs to see if it was due to a sudden increase in traffic. If not, I would look at the mysql slow query log to see if there were slow queries bogging up the database. More
  • At Cannes Covering the Film Festival
    You had probably noticed that the site was down for a couple hours on Monday and hadn’t been updated until this morning. I was on my way to the airport for Cannes and, of course, the site had to go down. More

 

Comment On This Story