Webserver - Spider Attack from several search engines at once - IT Security. This whole discussion possibly belongs over in Pro Webmasters, however; as pointed out in one of the comments is applicable here as well.
Having several search engines hit your site at once is absolutely normal behavior. If you managed to get links to your site from other reputable sites, you will get indexed. The most brutal are Yahoo, Yandex and Baidu. The first time our website bogged, I went into DDOS mode and got mostly embarrassed by finding every IP was coming from recognized Microsoft, Yahoo and Google sources (verified by IP block ownership, not UA Strings). The answer was to bolster the website resources to handle the load.
Despite what you surmize about the number of simultaneous users, your webserver must be able to handle Google, Bing and Yahoo simultaneously indexing your site plus your expected traffic. Verify where the traffic is coming from by analyzing your web server access logs. Today In Technology: The Official Blog of Canton Web Services. I normally do not complain publicly about other websites or companies, but in this case, I feel it is warranted.
On a busy forum that I administrate, I got an urgent message saying the forum was really slow. The following day, the complaint was that the forum was inaccessible for much of the afternoon. It did not take long to find out that one particular block of IP addresses (in the 180.76.0.0/16 range) had sent at least 120 searchbots to our forum during the brief time I checked it. At one point, our site logged 2,555 (!) Visitors, setting a record! In short, a classic Denial of Service (DOS) attack. Lesson? U.S. Government Targets Large BitTorrent Sites And Trackers. The US Government has classified some of the largest players in the BitTorrent scene as examples of sites which sustain global piracy.
Indexing and search engines The Pirate Bay, Torrentz, isoHunt, Kickasstorrents and BTjunkie all make appearances, with Demonoid, OpenBitTorrent and PublicBT described as trackers which have become "notorious for infringing activities. " In its “Out-of-Cycle Review of Notorious Markets”, the United States Trade Representative (USTR) has listed more than 30 Internet and offline physical ‘markets’ which it says exemplify “key challenges” in the fight against piracy and counterfeiting.
“Piracy and counterfeiting undermine the innovation and creativity that is vital to our global competitiveness. These notorious markets not only hurt American workers and businesses, but are threats to entrepreneurs and industries around the world,” said United States Trade Representative Ron Kirk. Heading the indexing list, as it does on so many occasions, is The Pirate Bay.
Denial-of-service attack. DDoS Stacheldraht Attack diagram.
Perpetrators of DoS attacks typically target sites or services hosted on high-profile web servers such as banks, credit card payment gateways, and even root nameservers. DoS threats are also common in business,[1] and are sometimes responsible for website attacks.[2] This technique has now seen extensive use in certain games, used by server owners, or disgruntled competitors on games, such as server owners' popular Minecraft servers. Increasingly, DoS attacks have also been used as a form of resistance. Richard Stallman has stated that DoS is a form of 'Internet Street Protests’.[3] The term is generally used relating to computer networks, but is not limited to this field; for example, it is also used in reference to CPU resource management.[4] Denial-of-service attacks are considered violations of the Internet Architecture Board's Internet proper use policy, and also violate the acceptable use policies of virtually all Internet service providers.