Hacked!

As email, documents, and almost every aspect of our professional and personal lives moves onto the “cloud”—remote servers we rely on to store, guard, and make available all of our data whenever and from wherever we want them, all the time and into eternity—a brush with disaster reminds the author and his wife just how vulnerable those data can be. A trip to the inner fortress of Gmail, where Google developers recovered six years’ worth of hacked and deleted e‑mail, provides specific advice on protecting and backing up data now—and gives a picture both consoling and unsettling of the vulnerabilities we can all expect to face in the future.
More

Against this assault, the Google security team, like its counterparts at other companies, is constantly monitoring activity across its systems, toward the end of detecting break-ins and hijacks before damage has been done, and even before the owners know that something has gone wrong. The signs they’re looking for have endlessly evolved: this year’s “Mugged in Madrid” message was last year’s “Mugged in London,” with enough minor changes to avoid many filters. “Google is a very data-centric place,” Eric Grosse, Google’s head of security, said to me. “We log lots of things and constantly have computers crunching over this data, trying to look for subtle patterns and keep up with the continual changes. That lets us do a better job of spam filtering than you could do on your own machine, since we see so much of it. We try to use the same techniques to look for other signs of abuse.” Jay Nancarrow, of Google’s communications team, added: “A lot of the signals we analyze are evident only because we’re offering the service to so many people. This is the kind of machine-learning the cloud enables that is very hard otherwise.”

I kept asking for examples of such tells; perhaps reassuringly, the Google officials kept saying they would rather not get into details. “But suppose we see that all your friends start marking mail from your account as spam,” Shreyas Doshi said. Obviously he didn’t mean that any human being at Google is monitoring your correspondence. Rather, its systems, like those of other companies that handle large numbers of online accounts, are constantly searching for anomalies. “Or if suddenly the recovery phone number in your account is changed to a number in South Africa. Or if you suddenly delete 20,000 e‑mails at once.” Any of these would be a sign that something might be going wrong, which Google would use to automatically immobilize the account. Indeed, the deletion of tens of thousands of e‑mails at once is what happened to my wife—and according to Gehring, the monitoring system had already noted an anomaly and was beginning to freeze the account by the time we filed our first report.

Perhaps the most startling thing I learned at Google about my wife’s case was how “lucky” we had been. Lucky not in having friends we could turn to in the otherwise automated and unapproachable Google edifice—though, of course, we were—but simply in the timing of the attack. If this had happened six months or a year earlier, or if it happened even today at most other e-mail services, the archives would likely have been gone forever. It was only because of the Undeletion Project that recovery, although slow, was feasible at all.

How could this be? How could big tech companies offer cloud services to hundreds of millions of people without better guarding their data against catastrophic loss? On Google’s side, one explanation involved complexities of the law. My wife and I might think that Google had a “duty” to be able to find her messages after some hacker had erased them. But according to Google’s legal department, its higher and more stringent duty is to ensure that messages are erased, if whoever is in charge of an account wants them gone. Political activists in repressive countries, people who for whatever reason (@RepWeiner) want parts of their electronic correspondence to disappear—they are the ones Google, like other e‑mail providers, had in mind in designing a system optimized for deletion rather than recovery. In exceptional cases, mainly in response to government orders in criminal or anti-terrorism investigations, Google could laboriously piece together already deleted records from its tape backups. But such recovery efforts were slow, hand-crafted undertakings, impractical for responding to thousands of hacking episodes a day (only some of which lead to total erasures).

“Every year, we do an immersion session with people involved in consumer operations,” I was told by Birendro Roy, one of the two young engineers at Google who developed the software for the Undeletion Project. He and his colleague Jaishankar Sundararaman were introduced to me as “the people who actually restored your wife’s mail.” (Roy, whose father is from India and mother is from Finland, grew up in Baton Rouge and went to MIT. Sundararaman grew up and went to college in Tamil Nadu, in southern India, and then did graduate work at Virginia Tech. I mention this as part of my ongoing chronicle of America’s strength through attracting foreign talent.) “The consumer reps tell us what are the common user ‘pain points,’ and restoring mail had definitely been something users were requesting.”

What a surprise, that people would want to recover from catastrophe! But from Google’s engineering perspective, the deleted-mail problem, while dire for those confronting it, affected only a tiny fraction of their users, and also was more complicated to solve than some other mainstream usability issues. In companies like Google, relatively few innovations are the result of top-down orders from executives. More evolve bottom-up, as engineers and product managers become sold on the need to add a new product or feature to the company’s offerings. Sometime last year, the Gmail engineers became sold on the value of a recovery system for maliciously deleted e‑mail. My wife’s luck was that her hack occurred this year rather than last—“Before, users were pretty much out of luck,” Bryant Gehring said—and that it hit the debugging cycle just when it did. To streamline a long story, the Undeletion Project software still had an important bug at the time of our first complaint. When the recovery software reached a message of a certain sort (“it was a ‘corner case’ involving saved drafts,” Sundararaman said), it mistakenly concluded that it had gotten to the end of the e‑mail list and could find nothing more. The engineers discovered the bug at just the time I was launching my “You cannot be serious!” screeds from outside. So now, if you lose your e-mail and start the recovery in time (deleted e-mail stays in a theoretically recoverable limbo for as long as 30 days, after which it’s likely gone for good), you should be able to get it back. In September, Google began offering a live help line for e-mail recovery, at a small cost (to deter abuse) of a few dollars.

“Of course, the very best way to recover data is not to lose it in the first place,” Shreyas Doshi said, in outlining what he described as a three-tier philosophy of data protection that was shared by Google, Yahoo, and other “cloud” companies. But that won’t help people like Deb when they make their horrifying discoveries. So Doshi outlined a tiered strategy I endorse for anyone who runs or uses e‑mail or Internet services—which is to say, everybody.

The first tier is for companies to give users a more realistic view of where and why they might be vulnerable, so they can do more to protect themselves. The second is for companies to reduce the incentive toward Internet crime by making it less profitable for the perpetrators—for instance, with better, faster ways of detecting hijackings, so that accounts will automatically be closed, even before the first “Mugged in Madrid” spam goes out. (And it seems that someone might usefully have a word with Western Union too.) The third is for companies to create better recovery systems, like the Undeletion Project, for when steps one and two fail.

Discussion of the cyber-threat menace has become a boom industry on its own. My latest immersion in the topic has left me more consoled than I might have expected—especially when I was imagining, early on, what the intrusion on my own household might mean to everyone else on the Internet who might innocently have exposed un‑backed‑up data to similar risk.

True, there is a lot to be worried about. “We have not examined a [corporate or governmental] network yet in which we have not discovered some level of intrusion,” Ron Ritchey, a longtime security expert now working at Booz Allen, told me. And “these things have been going on for years and years,” his colleague Ken Silva, the NSA veteran, said. “But companies are only now beginning to talk about them. After the Google report”—not about low-level cases like ours but about the massive Chinese intrusion on Google’s internal systems that the company revealed early last year—“the landscape for reporting these events really changed.” Both experts said the potential threat that concerned them most was not data theft or data loss, especially as it affected individual users, but data alteration that could affect institutions. “Imagine what could happen to the financial system, if trading data were altered,” Silva said. (Imagine: people might not trust financiers!) Or the medical data in a hospital. Or the formulas and data in spreadsheets on which companies base their bids and negotiations.

But on the other side, between us and data chaos are potentially protective factors of scale, technology, and strategy. Scale, in that big companies can invest big resources in monitoring and protecting their and their customers’ data. My money should be harder to steal if it’s in the bank than if it’s in my pocket, and something similar should be true of data. Technology, in that every person I spoke with emphasized that we’re living through an awkward stage in our current reliance on passwords. “Good passwords are bad for people, and bad passwords are good for criminals,” Michael Jones of Google said. Sooner or later, a better, easier, more reliable system of verifying identity will become widespread. It could be “biometric”—thumbprint recognition, iris scans, even a match of your face’s image, as you sit in front of the computer’s camera, with a stored picture of you. It could be purely digital, like Google’s recently introduced “two-step verification” system for Gmail identity, in which users enter both their password and a special code sent by Google to their cell phone or mobile device. None of these is perfect, but each one is a big advance over security now.

And strategy, in the form of a corporate and governmental emphasis on “data hardening” to limit the resulting loss when some attack inevitably succeeds. No free society can completely eliminate loss from crime, accidental death or destruction, even terrorism. The sane strategy is to prepare to contain the damage if and when things go wrong. Airbags in cars, watertight compartments in ships, hardened cockpit doors in airliners—these all illustrate the approach. The scary part in my wife’s case was that failure in one place meant potential failure of the system as a whole. Because she had used her password in one vulnerable place, all of the electronic data that mattered to her was at risk. If the data-hardening principle got through to me in the course of interviews, it has certainly occurred to security professionals at the institutions that collectively make up our new cloud info-system.

What about the rest of us, who are not security professionals? I asked that of every person I interviewed. Many of their recommendations boiled down to the hope that people would think more about their life online. “We’d like people to view their information life the way they view other parts of their life,” Andrew Kovacs of Google said. “It’s a good practice to review your financial situation every so often, and it’s a good practice to review your passwords and online-account information too.” Another official compared “cloud hygiene” to personal hygiene: you feel bad if you don’t brush your teeth or take a shower, and you should learn to feel bad if you’re taking risks online.

In practical terms, I have three action points I preach to anyone who will listen. Really, only two and a half.

The half step is extremely important, but I count it only as a half because it applies specifically to users of Gmail rather than other online systems. Here it is: if you use Gmail, please use Google’s new “two-step verification” system. In practice this means that to log into your account from any place other than your own computer, you have to enter an additional code, from Google, shown on your mobile phone. On your own computer, you enter a code only once every 30 days. This is not an airtight solution, but it can thwart nearly all of the remote attacks that affect Gmail thousands of times a day. Even though the hacker in Lagos has your password, if he doesn’t have your cell phone, he can’t get in.

In case you’ve missed the point: if you use Gmail, use this system. Also, make sure the recovery information for your account—a backup e-mail address or cell phone where you can receive password-reset information—is current. Google uses these to verify that you are the real owner.

Next we have password selection, that seemingly impossible task. The science, psychology, and sociology of creating strong passwords is a surprisingly well-chronicled and fascinating field. On The Atlantic’s Web site, we will describe some of the main strategies and the reasoning behind them. Even security professionals recognize the contradiction: the stronger the password, the less likely you are to remember it. Thus the Post-it notes with passwords, on monitor screens or in desk drawers.

But there is a middle ground, of passwords strong enough to create problems for hackers and still simple enough to be manageable. There are more details on our site, but strategies include:

• Choose a long, familiar-to-you sequence of ordinary words, with spaces between them as in an ordinary sentence, which more and more sites now allow. “Lake Winnebago is deep and chilly,” for instance. Or “my favorite packer is not brett favre.” You could remember a phrase like that, but a hacker’s computer, which couldn’t tell spaces from characters, would see only one forbiddingly long password sequence.

• Choose a shorter sequence of words that are not “real” English words. I once lived in a Ghanaian village called Assin Fosu. I can remember its name easily, but it would be hard to guess. Even harder if I added numbers or characters.

• Choose a truly obscure, gibberish password—“V*!amYEg5M5!3R” is one I generated just now with the LastPass system, and you’re welcome to it—and then find a way to store it. Having it written down in your wallet is one, though the paper it’s on shouldn’t say “Passwords” at the top. The approach I prefer, and use for some passwords, is to entrust them to online managers like LastPass or RoboForm. Even if their corporate sites were hacked, that wouldn’t reveal all your passwords, since the programs work by storing part of the encoding information in the cloud and part on your own machine.

At a minimum, any step up from “password,” “123456,” or your own birthday is worthwhile.

Finally, use different passwords. Not hundreds of different ones, for the hundreds of different places that require logins of some kind. The guide should be: any site that matters needs its own password—one you don’t currently use for any other site, and that you have never used anywhere else.

“Using an important password anywhere else is just like mailing your house key to anyone who might be making a delivery,” Michael Jones of Google said. “If you use your password in two places, it is not a valid password.”

I asked my experts how many passwords they personally used. The highest I heard was “about a dozen.” The lowest was four, and the norm was five or six. They all stressed that they managed their passwords and sites in different categories. In my own case, there are five sites whose security really matters to me: my main e‑mail account, two credit-card sites, a banking account, and an investment firm. Each has its own, good password, never used anywhere else. Next are the sites I’d just as soon not have compromised: airline-mileage accounts, Amazon and Barnes & Noble, various message boards and memberships. I have two or three semi-strong passwords I use among all of them. If you hacked one of them you might hack the others, but I don’t really care. Then there is everything else, the thicket of annoying little logins we all deal with. I have one or two passwords for them too. By making it easy to deal with unimportant accounts, I can concentrate on protecting the ones that matter.

At the end of my visit to Google, I went by to see Michael Jones, the friend to whom I’d first turned in data-loss despair. I told him what I’d learned, and how I would try to spread the message of shared responsibility, individual and organizational, for security in the cloud age.

“I see that you’ve got it!” he said. “The zeal of the convert. People in the business think about the risks all the time, but normal people don’t, until they’ve gotten a taste of the consequences of failure.”

I have now had that taste and am here to share the experience. As with so many other challenges in modern life, responding with panic or zealotry doesn’t get us anywhere. But a few simple self-protective steps can save a lot of heartache later on.

James Fallows is an Atlantic national correspondent; his blog is at jamesfallows.theatlantic.com.
Presented by

James Fallows is a national correspondent for The Atlantic and has written for the magazine since the late 1970s. He has reported extensively from outside the United States and once worked as President Carter's chief speechwriter. His latest book is China Airborne. More

James Fallows is based in Washington as a national correspondent for The Atlantic. He has worked for the magazine for nearly 30 years and in that time has also lived in Seattle, Berkeley, Austin, Tokyo, Kuala Lumpur, Shanghai, and Beijing. He was raised in Redlands, California, received his undergraduate degree in American history and literature from Harvard, and received a graduate degree in economics from Oxford as a Rhodes scholar. In addition to working for The Atlantic, he has spent two years as chief White House speechwriter for Jimmy Carter, two years as the editor of US News & World Report, and six months as a program designer at Microsoft. He is an instrument-rated private pilot. He is also now the chair in U.S. media at the U.S. Studies Centre at the University of Sydney, in Australia.

Fallows has been a finalist for the National Magazine Award five times and has won once; he has also won the American Book Award for nonfiction and a N.Y. Emmy award for the documentary series Doing Business in China. He was the founding chairman of the New America Foundation. His recent books Blind Into Baghdad (2006) and Postcards From Tomorrow Square (2009) are based on his writings for The Atlantic. His latest book is China Airborne. He is married to Deborah Fallows, author of the recent book Dreaming in Chinese. They have two married sons.

Fallows welcomes and frequently quotes from reader mail sent via the "Email" button below. Unless you specify otherwise, we consider any incoming mail available for possible quotation -- but not with the sender's real name unless you explicitly state that it may be used. If you are wondering why Fallows does not use a "Comments" field below his posts, please see previous explanations here and here.
Get Today's Top Stories in Your Inbox (preview)

When Will Robots Take Over the World?

"In a sense, we're already becoming cyborgs."


Elsewhere on the web

Video

Is Wine Healthy?

James Hamblin prepares to impress his date with knowledge about the health benefits of wine.

Video

The World's Largest Balloon Festival

Nine days, more than 700 balloons, and a whole lot of hot air

Video

The Origins of Bungee Jumping

"We had this old potato sack and I filled it up with rocks and dropped it over the side. It just hit the water, split, dropping all the stones. And that was our test."

Video

Is Trading Stocks for Suckers?

If you think you’re smarter than the stock market, you’re probably either cheating or wrong

Video

I Spent Half My Life Making a Video Game

How a childhood hobby became a labor of love

Writers

Up
Down

More in Technology

More back issues, Sept 1995 to present.

Just In