The majority of web traffic comes from the many robots that crawl, scrape, and otherwise skitter across sites. That's been the case since at least 2012. And a new report finds that while overall bot traffic dipped slightly in 2014, bots are getting sneakier.
The security firm Incapsula took a sample of some 15 billion human and bot visits over a 90-day period to better understand overall robot traffic on the web. It found that robots make up fewer site visits overall, 56 percent this year compared with 60 percent last year. But bots designed to cause mayhem are now outworking the bots that crawl for good—the ones behind RSS feeds, for example.
More traffic is coming from impersonator bots—rogue bots masked by proxy servers, data-stealing spy bots, bots that hide by making themselves look like legitimate search engine crawlers, and bots designed to knock out network access that have "browser-like characteristics." Incapsula estimates that traffic from this class of robots swelled nearly 10 percent since last year. It's the only bot category that's grown for three straight years, the firm said: "These numbers confirm what many security experts already know: Hacker tools are increasingly being designed for stealth." Bad bots of all kinds account for 29 percent of all website visits, whereas good bots account for 27 percent, Incapsula found.
Yes, bots can solve captchas, they can tweet, they can avoid cookies-based security measures. (They're really easy to deploy, too.) Bots also don't care how popular a site is—or, as Incapsula puts it, they're "hype immune." They simply go where they're told, and the Incapsula report suggests they're being told to go everywhere. Malicious bot traffic grows in an "almost exact proportion to a site's human traffic," meaning the overall risk factor for bot attacks is the same for large sites and small sites.
The fact that smaller websites get a larger proportion of their visits from robots—up to 80 percent of visits are from robots in some cases—may have more to do with the lack of security measures than anything else. For Incapsula's part, better security means users have to prove that they're human by explaining why they're visiting the site in the first place: "Bots can falsify their identities, but not their intentions."
We want to hear what you think about this article. Submit a letter to the editor or write to letters@theatlantic.com.
