@nixCraft Google did this back to one forum I was a part of years ago - and their stubborn refusal to obey robots.txt's crawl-delay directive still angers me to this day.

It's not just new crawlers that do this. Even the big old search engines have done it for ages.