How a Bunch of Hacked DVR Machines Took Down Twitter and Reddit

… and Spotify, and Github, and The New York Times

What began as a two-hour morning outage spanned well into the afternoon as Twitter, Reddit, Spotify, Github, and many other popular websites and services became effectively inaccessible for many American web users, especially those on the East Coast.

The websites were not targeted individually. Instead, an unknown attacker deployed a massive botnet to wage a distributed denial-of-service attack on Dyn (pronounced like dine), the domain name service (DNS) provider that they all share.

A distributed denial of service attack, or DDoS, is not an uncommon attack on the web, and web hosts have been fending them off for years. But according to reports, Friday’s attack was distinguished by its distinctive approach. The perpetrator used a botnet composed of so-called “internet-of-things” devices—namely, webcams and DVRs—to spam Dyn with more requests than it could handle.

Security researchers have been warning about these internet-of-things botnets since at least the summer. In September, a botnet composed of DVRs and CCTVs took down the blog of Brian Krebs, a prominent cybersecurity journalist. And on October 1, an anonymous developer posted source code online that allowed anyone to string a similar kind of botnet together.

Krebs wrote that releasing that software, called Mirai, “virtually [guaranteed] that the Internet will soon be flooded with attacks from many new botnets powered by insecure routers, IP cameras, digital video recorders and other easily hackable devices.”

The first of those attacks to be successful on a broadly destructive scale transpired on Friday.

“This feels new,” Bruce Schneier, a long-time computer-security researcher, told me by phone on Friday. “There hasn’t been a successful attack like this before.” There have been many unsuccessful ones that may have been larger, he added.

Andy Ellis, the chief security officer at Akamai, agreed. Akamai is one of the largest distributed cloud services on the web, serving between 15 and  30 percent of all web traffic. Some of its DNS products compete with Dyn’s.

“You never know how big an attack is on someone else,” said Ellis. He said this was a “watch-and-see” moment: Until Dyn describes the attack further, the security community would not know if this was an attack of unprecedented size or if it was one that had happened to find a specific weakness.

Neither Schneier nor Ellis would speculate about who might have perpetrated the attack.

“It could be orange elephants who became literate, for all we know,” Schneier said. “It might be three guys in Topeka.”

On his website, Krebs pointed out that a Dyn security researcher gave a talk on Thursday about the perils of internet-of-things botnets and the history of one DDoS mitigation firm in particular. Sometimes a retribution-style attack can follow a presentation of this type.

The attack demonstrates the fearsome power of internet-of-things botnets. Last month, Schneier argued in Motherboard that the government must regulate internet-of-things cybersecurity. “The market can’t fix this because neither the buyer nor the seller cares,” he wrote:

What this all means is that the [internet of things] will remain insecure unless government steps in and fixes the problem. When we have market failures, government is the only solution. The government could impose security regulations on [internet of things] manufacturers, forcing them to make their devices secure even though their customers don't care. They could impose liabilities on manufacturers, allowing people like Brian Krebs to sue them. Any of these would raise the cost of insecurity and give companies incentives to spend money making their devices secure.

Ellis struck a less apocalyptic tone when he described the situation on the phone to me.

“Historically, when you see new attack capabilities show up —in volume or type of attack—you see some outages, then you see people adapting, then people make the investments needed to scale up infrastructure,” he said.

This isn’t even the first immobilizing attack on a DNS server ever. On the morning of June 15, 2004, a DDoS assault on Akamai’s DNS servers effectively blocked access to the websites of Apple, Google, Microsoft, and Yahoo. That outage did not last as long as Friday’s assault, though.

DNS is also especially vulnerable to a sustained attack, because DNS providers don’t necessarily update their records in real time. On Friday, for instance, Github changed its DNS provider so that its visitors would be rerouted to a new server. But it will take until Saturday or Sunday for that change to propagate across the internet.

Ellis said that some companies might react to the attack on Dyn by using many different DNS name servers at once—so that if one came under attack, others would take its place—many will “weather it out and wait to see what Dyn will do,” Ellis said.

“The internet isn’t down,” he added. “Packets are still getting through.” Only one DNS provider was ever blocked, he said. The rest of the infrastructure still work—even if Twitter, Reddit, Spotify, and the Times were all, for a time, essentially inaccessible.

Ellis told me to look on the bright side: “Productivity is up, because all the things that people use to procrastinate at work are down.”