Log in

No account? Create an account
durka durka
[Most Recent Entries] [Calendar View] [Friends View]

Wednesday, October 27th, 2004

Time Event
MSNBot was killing my server
Looking through the logs on my server, I realized that MSNBot and Googlebot had been beating the hell out of it for the past two months. Bots tend to take their time crawling a site, but these just followed every link going back through all images on my lj images site and loading every fridge on my fridge poetry site. It wasn't really polite about it, either. Since I've put up these sites, their bots have pulled somewhere close to 1Gig of traffic from my site and over 50,000 hits from these bots. That's just not polite.

I've since configured robots.txt files to keep unwanted crawlers out and keep my database from being pounded for no good reason. Now, if you google for "site:poisonkitty.net poetry images" you get just about every kitchen sink that my site ever had to offer. Also, since one of my sites gets a feed from Slashdot, every summary of every slashdot article is indexed. It points to my site, which points to slashdot.

Geez, why didn't I realize this sooner?

<< Previous Day 2004/10/27
Next Day >>
PoisonKitty Poetry   About LiveJournal.com