Bandwidth Issues with Google Bot 2

[danima]

Hey folks:

Our company is experiencing a big problem with bandwidth on a few of our key clients' sites. When trying to figure out why these sites had huges spikes in bandwidth over the past 3 or 4 months one of our programmers noticed that a huge portion of the bandwidth was coming from crawl-66-249-##-##.googlebot.com and crawl-#.cuill.com

I'll admit...this has thrown me for a loop because it's just the last several months. Some of these sites have been up for a few years?

Any insight into this problem or solutions or even some direction as where to look with be very much appreciated.

Thanks and hopefully the above makes sense. If I need to send links...I will.

 

[WhoKilledKenny]

Suggestions:
Verify they are actually google bots...
There is no doubt that a ton of bot activity on one's sites are from rogue spiders. Spider or bots that pretend to be legit bots but are there to steal your content. We have covered several sessions on this in the past; here are some:

http://www.seroundtable.com/archives/012616.html

If you are using Google webmaster tools, you can specify the crawl rate. Faster crawl rates put more load on your servers.


Jesse Hamrick

http://www.powershellpro.com

 

[danima]

Makes sense....I have to admit...I'm pretty lax when it comes to creating robot.txt files...I'm assuming this will help too?

 

[ChrisHirst]

For anything to do with Twiceler (the cuill bot) email them. They are very helpful if you have problems with the bot.

http://www.cuill.com/contact.html

Google have been on a massive recrawl over the past few months. A "new Google" was rolled out earlier and with the subsequent dumping of the "Supplemental" index they are on a mission to bring their indexes up to date and many sites are getting "deep crawls"

Crawling too fast.

http://www.google.com/support/webmasters/bin/answer.py?answer=33571&topic=8460

Chris.

Indifference will be the downfall of mankind, but who cares?
Woo Hoo! the cobblers kids get new shoes.
People Counting Systems

So long, and thanks for all the fish.