Poor robots. The really good ones just can’t seem to keep up with the growing number of humans and bad robots on the web.
That was the main finding of a new report by the security firm Imperva, which is set to release its annual survey of Internet-bot traffic on Thursday. Good robots are the helpers: They’re designed to crawl the web for archiving purposes, to populate RSS feeds and search engines, and to help with other automated tasks that assist humans. Bad bots, by contrast, are the impersonators, data thieves, and other rogue agents.
Up until now, in the four years that Imperva (formerly Incapsula) has been studying web traffic, bots had always driven the majority of web activity. But this year, there was a surprise. “We saw a changing of the guard, with humans stepping in to become the Internet’s new majority,” the companyinc wrote. In 2015, humans were responsible for nearly 52 percent of all online traffic. Two years ago, humans drove less than 39 percent of overall web traffic.
This apparent reversal turns out to be more nuanced than it appears at first. The volume of Internet activity by good bots, relative to humans and bad bots, declined—but did so in part because human online activity grew in 2015, and bad bot traffic stayed steady compared with last year. “Basically, what we had here was a case of good bots collectively not keeping up with growth of human and bad bot traffic,” Imperva wrote.
But there’s more: On popular websites that get the most overall traffic from humans, the share of good-bot activity dropped significantly, from 22 percent to 9 percent. But among the sample of websites that logged fewer than 100,000 human visits a day, good bot traffic went up. “It appeared that, the more popular a website got, the harder it was for the good bots to keep up with the influx of human and bad bot visits,” Imperva wrote.
Why is that?
For one thing, good bots aren’t enticed by the same factors that make a website popular to humans. “In fact, the more popular a website gets, the faster its human population tends to grow,” Imperva said. And although it seems reasonable to expect a popular website would draw more scammer bots, “the amount of extra sessions generated are negligible.”
The backdrop for all this web activity is a population of human Internet users that’s still growing dramatically. The number of people who used the Internet this year exceeded 3.1 billion, up from 2.9 billion last year. And individuals continue to spend longer periods of time using the Internet, too. “The compound effect of these growth trends, and the fact that only humans are motivated to consume and share popular content, are the reason why good bots—who lack the same motivations—simply can’t keep up,” Imperva wrote. “Unlike good bots, malicious non-humans are employed by individuals, not organizations. As a result, their growth is closely tied with the increase in Internet human population.”
“In absolute terms,” the report said, “the sheer amount of bot and human traffic on our network is always increasing.”