Mark (poisonkitty) wrote,

MSNBot was killing my server

Looking through the logs on my server, I realized that MSNBot and Googlebot had been beating the hell out of it for the past two months. Bots tend to take their time crawling a site, but these just followed every link going back through all images on my lj images site and loading every fridge on my fridge poetry site. It wasn't really polite about it, either. Since I've put up these sites, their bots have pulled somewhere close to 1Gig of traffic from my site and over 50,000 hits from these bots. That's just not polite.

I've since configured robots.txt files to keep unwanted crawlers out and keep my database from being pounded for no good reason. Now, if you google for " poetry images" you get just about every kitchen sink that my site ever had to offer. Also, since one of my sites gets a feed from Slashdot, every summary of every slashdot article is indexed. It points to my site, which points to slashdot.

Geez, why didn't I realize this sooner?
  • Post a new comment


    default userpic

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.