Internet, Robots.txt | featured news

Relieving High Server Load by Blocking Search Bots

Coding

Over the years, whenever the site was slow to load up, I always went through the same routine. I checked traffic logs to see if it was due to a sudden increase in traffic. If not, I would look at the mysql slow query log to see if there were slow queries bogging up the database. If that wasn’t the culprit, then I would take a look at the access logs to see if there were any irregular activities from search engine bots.

 

Looks like Murdoch’s just started blocking search engines

Looks like Murdoch’s just started blocking search engines

Remember when Rupert Murdoch caused a stir by saying that he was going to start blocking news search services like Google News from carrying his sites’ stories? Well, it looks like he’s started.

 

Subscribe to this RSS topic: Syndicate content