The majority of web traffic comes from the many robots that crawl, scrape, and otherwise skitter across sites. That's been the case since at least 2012. And a new report finds that while overall bot traffic dipped slightly in 2014, bots are getting sneakier.
The security firm Incapsula took a sample of some 15 billion human and bot visits over a 90-day period to better understand overall robot traffic on the web. It found that robots make up fewer site visits overall, 56 percent this year compared with 60 percent last year. But bots designed to cause mayhem are now outworking the bots that crawl for good—the ones behind RSS feeds, for example.
More traffic is coming from impersonator bots—rogue bots masked by proxy servers, data-stealing spy bots, bots that hide by making themselves look like legitimate search engine crawlers, and bots designed to knock out network access that have "browser-like characteristics." Incapsula estimates that traffic from this class of robots swelled nearly 10 percent since last year. It's the only bot category that's grown for three straight years, the firm said: "These numbers confirm what many security experts already know: Hacker tools are increasingly being designed for stealth." Bad bots of all kinds account for 29 percent of all website visits, whereas good bots account for 27 percent, Incapsula found.