The warnings began trickling in around supper time on Wednesday night.
AT&T was experiencing what officials in several states described as a nationwide outage affecting 9-1-1 services. The problem was that some AT&T customers who dialed 9-1-1 in at least 18 states couldn’t reach emergency dispatchers—and instead would hear either a busy signal or a phone that kept ringing and ringing and ringing and ringing and ringing.
It was nearly three hours before public officials gave the all clear that the issue had been resolved.
AT&T declined repeated requests for information about the scope and cause of the outage, but local officials acknowledged potential problems in Alabama, Arkansas, California, Colorado, Florida, Indiana, Kansas, Kentucky, Louisiana, Maryland, Massachusetts, North Carolina, Nevada, Pennsylvania, Tennessee, Texas, Virginia, West Virginia, and Washington, D.C.
Now, the Federal Communications Commission says it is investigating what happened.
The outage was stunning in part because of its scope, but also because of how reliable 9-1-1 services usually are. One massive failure, however, is all it takes to raise a host of serious questions about the integrity of a critical public-safety system in the United States.
“The reliability of the system is really astonishing given the age of some of the components,” said Trey Forgety, a cybersecurity expert and the director of government affairs for the National Emergency Number Association, a professional organization focused on policy, technology, and operations related to emergency communications in the United States.
The patchwork of switches, routers, and cables that are now involved in completing a 9-1-1 call includes bits of technology that are no longer manufactured. One widely used component was just discontinued last year, Forgety says.
“The switches that we call selective routers—those devices in some cases date back to the early 1980s,” Forgety told me. “These are things you can’t get parts for anymore. Some of these switches, if you ever lost one in its entirety—if you had a fire, or a flood, or a terrorist attack—they’re not actually replaceable.”
Outdated switches are unlikely to have played a role in the AT&T outage, however. The scuttlebutt among leaders in the emergency-response industry was that a scheduled or automatic network-configuration change by one of AT&T’s vendors was to blame. AT&T did not respond to a request to confirm this characterization of the outage.
Today, just routing a 9-1-1 call involves multiple companies, which are often geographically remote from where the call is placed. Yet placing an emergency call is still “pretty darn reliable,” says Roger Hixson, NENA’s director of technical issues. “Practically everything is duplicated so that if one piece fails, the rest of the system can handle 9-1-1 calls pretty seamlessly,” he told me. “Very seldom do we have a case where there’s a failure that actually affects the ability of calls to get to 911 centers.”
Very seldom isn’t never. In 2006, a computer glitch caused an 9-1-1 outage that lasted for nearly seven hours in Pittsburgh. The death of at least one child was tied to the outage, during which time a father tried for 19 minutes to call for help when he found his infant son not breathing. More than 200 other calls to 9-1-1 went unanswered during the outage, according to news reports at the time.
The FCC said in 2014 it would prioritize reversing a growing “trend of large-scale ‘sunny day’ 911 outages,” a reference to system failures caused by software and database errors rather than bad weather, which was more of a problem in the days of analog telephone. “It’s more complex today than it used to be,” Hixson said.
The United States began implementing its universal emergency telephone line nearly 50 years ago. Incidentally, it was AT&T that first announced, in 1968, that it would use the digits 9-1-1 as the nation’s emergency code. 9-1-1 was selected because it was short, easy to remember, and not a sequence already in use.
But it was several decades before 9-1-1 was a near-nationwide service. Not even one-quarter of the U.S. population had access to the service by the mid-1970s. By the late 1980s, about half of Americans could dial 9-1-1 in an emergency. Today, about 96 percent of the United States is covered by 9-1-1, according to NENA.
While many more people can use 9-1-1 today, a system failure like the one that occurred on Wednesday night would have been practically unheard of even two decades ago. “It would have been difficult for anything to happen nationwide 20 years ago, because these systems are basically localized in parts of states,” Hixson said. “Now we have nationwide carriers. If they’re not careful about how they do certain things, or how much duplication redundancy they have, it becomes more difficult to manage.”
More difficult still because up to three-quarters of 9-1-1 calls now come from mobile devices, Hixson told me. New technology has introduced several new potential vulnerabilities. The paradox is that while 9-1-1 systems are likely more reliable than ever, there are more points at which things can go wrong.
Along with wireless network outages, there are distributed denial of service attacks to worry about—the kind of attack against a call center’s computer system that would flood dispatchers with fake calls, making all the phones ring at once. There’s also the possibility of an attack that would manipulate key information that goes out to emergency responders—a way of making them show up at the wrong house, for example.
“That’s a significant threat,” Forgety told me. “The worst one we can imagine is if some malicious actor wants to undertake an act of terrorism and hamper the local response to that [attack]—disrupting 9-1-1 communications entirely.”
Ironically enough, the aging systems first responders use to communicate with one another may offer a layer of protection in such a scenario. “Those radio systems typically use old-school technology,” Forgety said. Which makes them safe from computer-age attacks. Then again, he added, “nothing is impossible.”
