Hacked!

As email, documents, and almost every aspect of our professional and personal lives moves onto the “cloud”—remote servers we rely on to store, guard, and make available all of our data whenever and from wherever we want them, all the time and into eternity—a brush with disaster reminds the author and his wife just how vulnerable those data can be. A trip to the inner fortress of Gmail, where Google developers recovered six years’ worth of hacked and deleted e‑mail, provides specific advice on protecting and backing up data now—and gives a picture both consoling and unsettling of the vulnerabilities we can all expect to face in the future.
More

On April 13 of this year, a Wednesday, my wife got up later than usual and didn’t check her e‑mail until around 8:30 a.m. The previous night, she had put her computer to “sleep,” rather than shutting it down. When she opened it that morning to the Gmail account that had been her main communications center for more than six years, it seemed to be responding very slowly and jerkily. She hadn’t fully restarted the computer in several days, and thought that was the problem. So she closed all programs, rebooted the machine, and went off to make coffee and have some breakfast.

When she came back to her desk, half an hour later, she couldn’t log into Gmail at all. By that time, I was up and looking at e‑mail, and we both quickly saw what the real problem was. In my inbox I found a message purporting to be from her, followed by a quickly proliferating stream of concerned responses from friends and acquaintances, all about the fact that she had been “mugged in Madrid.” The account had seemed sluggish earlier that morning because my wife had tried to use it at just the moment a hacker was taking it over and changing its settings—including the password, so that she couldn’t log in again. The bogus message that had just gone out to me and everyone else in her Gmail contact list was this:

From: Deb Fallows <debfallows@gmail.com>

Date: Wed, Apr 13, 2011 at 8:45 AM

Subject: Problem

To:

now this might come as a suprise to you,but I made a quick trip to Madrid in Spain and was mugged.My bag,valuables,credit cards and passport all gone.The embassy has cooperated by issuing a temporary passport.I need funds to settle outstanding hotel bills,ticket and other expenses.

To be honest,i don’t have money with me at the moment. I’ve made contact with my bank but the best they could do was to send me a new card in the mail which will take 2-4 working days to arrive here from DC. I need you to lend me some Money to sort my self out of this predicament, i will pay back once i get this over with because i need to make a last minute flight.

Western Union or MoneyGram is the fastest option to wire funds to me. Let me know if you need my details(Full names/location) to effect a transfer. You can reach me via hotel’s desk phone and the number is, +34 981 600916867.



Thanks.

Deb

I’m using her real e‑mail address because at this stage there’s no point in “protecting” it. Someone had obviously taken over her account and was using it as a crude spamming tool—or at least what we considered crude. Who hadn’t seen countless messages like this before? Which of her friends would really think that Deb would capitalize “Money,” type a paragraph’s worth of sentences with no spaces separating them, or say that she had gone to “Madrid in Spain”? And, indeed, the great majority of notes were warnings that her account had been hacked and was being used to send out fraud-spam; some included gratuitous tips about the need to be more careful in online life.

But a touching handful of the notes came from people who took the plea at face value. They wrote to me; they replied to Deb’s account; a few even phoned from South America, Asia, the Midwest, and Australia to find out how they could help. That was the first indication that this would be more than a minor nuisance. What if some of them actually sent cash?

The more serious sign of the potential scale of our problems came later in the day. Google offers a variety of automated ways for users to regain control of Gmail and other accounts they think have been hacked. The automated routines, plus an online forum moderated by Google employees, are the only help Google offers. With hundreds of millions of active Gmail accounts to manage—that’s as specific as Google will be about its user base—operating in 54 languages worldwide, the relative handful of human beings on Gmail’s support staff could not even pretend to offer live one-on-one service. The same is true of Yahoo, Microsoft’s Hotmail, Facebook, Skype, eBay, and the other big operators of “cloud”-based systems.

As a reminder: in cloud-based systems, users turn the management and protection of crucial data and services over to third parties, and then call up information as necessary via the Internet. For individuals, the appeal is that e-mail held “in the cloud” by Google, Yahoo, Microsoft, et cetera, is available wherever there is an Internet connection, rather than being lodged on any one machine. If a corporation is large enough, it may operate its own internal cloud, or turn to large-scale suppliers like Amazon—which has a cloud-server business apart from its familiar retail functions—to store and protect data.

The first and easiest automated step was to fill out a password-reset form. Doing so prompted Google to send reset instructions to the mobile-phone number or alternate e‑mail address listed as “recovery options” for Deb’s account. That alternate e‑mail account, with AOL, was no longer active, and in any case whoever had taken over her Gmail settings seemed to have removed or changed the information. The next line of defense was to submit a form reporting that an account had been taken over or compromised. We had sent in that form within 30 minutes of discovering the problem, giving my Gmail address as the new contact point. Meanwhile my wife logged into a secondary Gmail account she had previously created, and began writing to friends and family as quickly as she could to explain what was going on.

While we waited for results from Google, we began to hear, by phone and via our other e‑mail accounts, from people who had written back to Deb “in Madrid” to find out more about her predicament. They had all quickly gotten responses, from an account meant to look similar to hers but with a one-letter difference: debbfallows@gmail.com. We learned later that, as a predictable part of a hacking attack on Gmail, Yahoo, Hotmail, or any other e‑mail service, an attacker will change the settings so that all incoming mail is forwarded to a new, similar-seeming address—and then deleted from the real account, to make it harder for the real user to know, later on, who had responded to the scam. And whoever was on the other end of the exchange had gathered at least enough personal info to sustain a round or two of exchanges with concerned friends. For instance, Deb’s mother in Florida, 89 years old, had written back immediately to say that the message was a fraud, signing the note “Mom.” Her message went instead to debbfallows, who sent this reply:

From: “Deb Fallows” <debbfallows@gmail.com>

Date: Wed, 13 Apr 2011 14:23:23 +0000

Subject: Re: terrible scam

Hey mom,

I am too old to raise a false alarm. I was mugged last night under gun point.



I need €1,500. Below are my details you for the transfer,kindly get this done from any Western union office close to you:



Receiver Name: Deborah Fallows

Location: Madrid,Spain



Don’t forget to email the Money control number(MTCN) to enable me pick up the funds. I promise to pay back once I return to DC. Expecting to hear from you.



Other than this,how are you doing?



Xoxo,

Deb

We thought that “other than this” was a nice touch. Other friends who replied got other variations, all with the same basic social-engineering info—the knowledge that our regular home was “in DC,” and that I was her spouse. A friend in New York was ready to send money, if he could only talk with Deb on the phone to be sure. His exchange with the hacker finally petered out this way:

I had to check out of the hotel due to accumulated bills. I am in a cyber cafe at the moment,sorry I cannot afford a call card.



Xoxo

Deb

So far, this was embarrassing, and possibly costly to the most openhearted or trusting among our acquaintances, but not worse than that. We’d returned only a few days earlier from a two-month stay in China. Perhaps this was one more predictable aftereffect, like my chronic cough? Things seemed to be improving when, around 2 p.m., a message from Google’s help system arrived in my account, with instructions on how Deb could at last reset her password and regain control of her information.

She did so, and logged into her Gmail account with enormous relief, which lasted perhaps five seconds. When she looked at her Inbox, and her Archives, and even the Trash and Spam folders in her account, she found—absolutely nothing. Of her allocated 7 gigabytes of storage, 0.0 gigabytes were in use, versus the 4+ gigabytes shown the day before. Six years’ worth of correspondence and everything that went with it were gone. All the notes, interviews, recollections, and attached photos from our years of traveling through China. All the correspondence with and about her father in the last years of his life. The planning for our sons’ weddings; the exchanges she’d had with subjects, editors, and readers of her recent book; the accounting information for her projects; the travel arrangements and appointments she had for tomorrow and next week and next month; much of the incidental-expense data for the income-tax return I was about to file—all of this had been erased. It had not just been put in the “Trash” folder but permanently deleted.

In some other circumstances, we might have had a calmly reasoned discussion about whether it made sense to have so much emotionally and practically precious information in a single, now evidently vulnerable, place. Even in these real circumstances, we realized that with enough persistent effort she could have eventually rewoven parts of the missing fabric. Her mother still had some of the messages Deb had sent about her father, I had some of what she’d written and done in China; bit by bit she might get some things back. For the moment, all we could do was clean up some of the traces of the attack that remained in her account—the command to forward all incoming messages to DebbFallows, the bogus e‑mail address and phone numbers for the password-recovery routines—and fill out another form on the Google help site, this one to request an automated recovery of deleted e‑mail.

It was at about this time that I started thinking about the ramifications of this problem beyond our own situation, desperate as that situation felt to us just then. Through more than 30 years of computing, I’d had my ups and downs with data storage. My very first computer, a Processor Technology SOL-20, was nearly incinerated along with all of its electronic contents when a lightning bolt hit our house in the early 1980s. (The contents included the notes and drafts for my book National Defense, which fortunately I’d printed out on paper.) Hard disks fail; laptops get dropped. But I’d never before imagined the chance of total, catastrophic, years’ worth of loss. This was a loss whose sweeping magnitude was possible only because my wife had entrusted her data exclusively to the most professional of pros: Google’s operation in the cloud. If we had thought that data security was strictly up to us, we’d have made backups of some sort to limit the potential damage—much as we would lay in our own firewood and keep our own chickens and cows to be sure we’d never freeze or starve if normal supplies were cut off. In my own version of Depression-style thinking, and with that lightning strike in mind, I had always made triply redundant backups of anything that mattered to me, including e‑mail. Local on-disk backups of Gmail archives, via programs like Eudora and Thunderbird—or both. Online backups of those local backups, through SugarSync and Dropbox—and then more local backups on my other machines. But my wife had trusted the cloud and Google. And now?

Her move to the cloud had coincided with the larger and irreversible shift of business, personal, governmental, and every other sort of activity to the cloud. The shift is irreversible because it brings so many advantages. Who would go back to searching for addresses on paper maps after using online mapping services? Needing to save and file canceled paper checks rather than inspecting them online, or doing a thousand other chores in pre-cloud form? In addition to these corporate and public services, whose users are increasingly conducting their business and storing their data in the cloud rather than on paper, our personal data has moved to the cloud as well, with the premise that we’ll be able to retrieve and work on our correspondence, our contacts, our photos and documents, from any computer connected to the Internet. But, of course, the more we rely on the cloud, the more we expose ourselves to its vulnerabilities. These include the breakdowns that affect any complex system. When much of Washington had a multiday power outage after a snowstorm last January, the loss of Internet service seemed almost as crippling as the loss of light and heat. They also include deliberate attacks—for criminal gain, spying, or sabotage—that are sure to increase as the value of cloud-based information does. “Where the money is, that is where the criminals will go,” a former National Security Agency official named Ken Silva, who now works as an online-security specialist for Booz Allen Hamilton, told me this summer. “Where the sensitive information is concentrated, that is where the spies will go. This is just a fact of life.” The more important online storage becomes, the more relentlessly it will be under attack.

For instance: Chastened by my wife’s experience, I decided to make my online passwords “stronger,” and to shift to an online storage site to manage them. The following week, that site—LastPass.com—was itself hacked and some of its data stolen. (I still use it, as I’ll explain.) At around the same time, the anonymous hacker group LulzSec, operating under the motto “Laughing at your security since 2011” (the first part of the name is phonetic for “LOLs”; the second stands for “security”), started functioning as a kind of tech-world version of WikiLeaks, penetrating corporate sites and then publishing large numbers of usernames and passwords.

Sony, Citibank, Veterans Affairs, major hospitals, tech firms like Intel, Cisco, and Google—I stopped keeping track of the institutions that announced intrusions, after security experts told me that essentially every major organization suffers ongoing attacks. But I used the shock of my wife’s experience as an occasion to educate myself about the vulnerabilities and new rules of operation in the cloud era, as they involve corporations and institutions as well as individuals. What I found is not all good news, but it is better than I might have feared. It includes some hopeful signs about the way corporations and governments are defending their data, and manageable practical steps individuals can take to avoid scares like the one my wife had that day.

I say “scare” rather than “trauma” because—to skip ahead in the story—my wife eventually got her e‑mail back, through Google’s recent “Undeletion Project,” as I called it when I learned of it. But it was a long time before that happened, and our attitude toward Google got much worse before it got better. I concentrate on Google here because that’s where we had our problem, and more generally because of its exceptional international role. But everyone I spoke with there and at other organizations emphasized that our experiences with Gmail—the brush with disaster and subsequent revelation of the gulf between data professionals’ view of reality and what the rest of us assume—were not exceptional at all but were variations on a cloud-wide theme. And our experience and revelation would apply to most people using most online services, including Apple’s pending “iCloud” services and Microsoft’s continuing movement of Windows services to the cloud.

I felt antsy rather than sleepy on that first night after the attack, as I kept fielding calls and e‑mails from friends and spending time on hold trying to change our credit-card numbers. So I was still at the computer a little after 2 a.m., monitoring both of our e‑mail accounts, when Google’s recovery team sent its response to our “My e‑mail is missing” form. I’ve boldfaced the parts that jumped out at me:

From: The Google Team

Date: Thu, Apr 14, 2011 at 2:01 AM

Subject: Re: [#791225671] (no subject)

To: debfallows@gmail.com



Hello,



We have processed your request to recover mail that may have been inappropriately purged from your Gmail account. Any previously deleted messages that we were able to recover will now be in your account in a newly added label called ‘recovered <time stamp>.’ If the message(s) you are looking for are not in this label, they unfortunately are not recoverable.

If you have not already done so, we suggest that you take the steps outlined in our Security Checklist to ensure the security of your account: http://mail.google.com/support/bin/static.py?page=checklist.cs&tab=29488

We unfortunately will not be able to respond to any further emails on this case.



Sincerely,

The Google Team

I looked immediately at the “recovered <April 14>” folder. A little over a thousand messages were there, reaching back to January of this year. But from the preceding six years, nothing at all. And that was it? “Unfortunately” they are not recoverable and Google “will not be able to respond to any further emails on this case”?

I waited until my wife got up a few hours later to rant and rave and yell. A company presents itself as the world’s leader in handling big data; it attracts users to its services (albeit free, but indispensable to the company as advertising vehicles) with the idea that trusting cloud services is moderne; and then it exposes them to something few sane users would accept on their own—the risk of “single-point failure” that could in a few minutes eliminate many years’ worth of crucial data. This is the same company famed for making every bit of data part of the world’s “permanent record.” That embarrassing picture of you in a nightclub, that subversive definition of “santorum”—they and other ephemera are eternal, but all your e‑mail can disappear before noon?

At this point I thought: To hell with journalistic detachment. Over the years I’d come to know lots of people at Google, and I decided to forward the “Unfortunately …” message to one of them. This friend was not Eric Schmidt, the company’s longtime CEO and now executive chairman, whose family my wife and I had gotten to know long before his Google era. (Embarrassingly enough for us, and possibly for him, he had received one of the “Mugged in Madrid” notes, which he passed on to me with a terse “Deb’s e‑mail has been hacked” subject line.) But the person I sent it to, Michael Jones, Google’s “Chief Technology Advocate,” was in a position to direct extra attention to the problem. My message was: You (Google) cannot be serious about this. You cannot entice people into relying on your services, and be so cavalier about the risks they are exposed to. Can you?

A little more than a week later, after several more warnings that “unfortunately” nothing might be recoverable, my wife did in fact get her messages back, all 4+ gigabytes. The first thing I did was to back them all up onto her hard disk, with Thunderbird—and then back up those archives elsewhere, just in case. But one of the next things I did was to arrange a trip to Mountain View to try to understand what had happened. My main discoveries exposed the gulf between the way information professionals understand the realities, vulnerabilities, and responsibilities of the cloud era and the way the rest of us do.

After interviews at Google with staff members ranging from the senior officials who set security policy to the young engineers who had eventually figured out how to recover maliciously deleted e‑mail (including my wife’s), plus follow-up interviews elsewhere, I had three “key takeaways,” as they say in the tech world. They involved the scale of the hacking problem, not just for individual users like us but for organizations; the nature of the arms race between people trying to steal or alter data and those trying to protect it; and the expectations of what citizens need to do to protect themselves.

At Google I asked Byrant Gehring, of Gmail’s consumer-operations team, how often attacks occur. “Probably in the low thousands,” he said. “Per month?,” I asked. “No, per day,” followed by the reassurance that most were short-lived “hijackings,” used to send spam and phishing messages, and caused little or no damage, unlike our full-out attack. My wife and I, having heard from half a dozen friends who’d recently had similar problems, had innocently imagined that we all were part of some general upsurge in Gmail attacks. In our grandiosity, we thought it was perhaps even aimed at journalists. But according to the experts, while there are more e‑mail attacks worldwide than a year ago, it was mere coincidence that people we knew had been hit around the same time. On average, half a dozen accounts are taken over every two or three minutes, round the clock, including now.

Why are so many accounts so vulnerable? Again we were naive in the assumptions we’d made about our own case. We’d just returned from China, where everything about Internet use in general and Google services in particular is insecure and fraught. Had some malware made its way onto our computers? What about that time my wife entered her Gmail username and password on a public terminal at the airport in Zhuhai? Might it have had a keystroke logger, recording names and passwords for later misuse?

Perhaps it did, but that didn’t seem to be our problem. As in the great majority of hacking cases, my wife had been using the same password for her Gmail account as for some other, less secure sites, where her username was her Gmail address. (Who hasn’t done this?) And one way or another, a list of e‑mail addresses and associated passwords from one of those sites had made its way to hackers. A possible source was the notorious Gawker hack of 2010, when more than a million e‑mail addresses and matching passwords for people who had registered on Gawker sites were stolen. “If you have ever used the same password in more than one place, you have reduced your overall safety record to whichever site had the lowest amount of protection,” Jones told me during my visit. Yet this is an overwhelmingly common practice. An analysis of posted username/password combinations after several recent hacks showed that in two-thirds of the cases, if you knew a user’s password for one site, you knew it for another.

It’s possible, too, that my wife’s password was simply “guessed,” though in a different way from what laymen might assume. Guessing less often involves social engineering—trying your birthday or your hometown or your relatives’ names—than “brute-force attacks,” in which a hacker’s computer tries every word or combination of words in existence, in a variety of languages, to see if it finds a match. From most officials, I heard reminders that if a password can be found in a dictionary, that password is not safe. Andrew Kovacs, communications manager for the Google security staff, added: “And those tricks about changing E’s to threes and O’s to zeros? Sorry to tell you, but the hackers have thought of those too.” Several of the people I spoke with pointed out that brute-force attacks have recently become much more effective, as hackers have taken advantage of the powers of new computer-graphics chips, which can handle certain kinds of computations even more quickly, and with more parallel processes running simultaneously, than a computer’s central processing chip can. These turn out to be the computations necessary for producing password hacks.

My wife’s password was judged as “strong” when she first chose it for use with Gmail. But it was a combination of two short English words followed by numbers, so if it didn’t leak from some other site, it might just have been guessed in a brute-force attack. For reasons too complex to explain here, even some systems, like Gmail’s, that don’t allow intruders to make millions of random guesses at a password can still be vulnerable to brute-force attacks.

Once her password got out, it ended up, according to later sleuthing by officials at Google, in the hands of someone possibly in Cote d’Ivoire but probably in Nigeria. My wife and I could again flatter ourselves that we were the objects of Chinese state security, but the probable truth is more mundane—“it’s like being the victims of pickpockets,” as Jay Nancarrow, of the Gmail team, put it. Why Nigeria? All of the people I spoke with had thought about this question—“I think about it every day,” Bryant Gehring said—and no one had a fully convincing answer. Their hypotheses involved some combination of talent, opportunity, and lack of legal controls. “They have the Internet, they can get rich this way, there’s not really a lot of government enforcement,” Gehring said. “For a lot of crafty people, it can be the best way to make a living.” That could apply to a lot of places, but it seems to be most relevant to West Africa.

In “The Chilling Story of Genius in a Land of Chronic Unemployment,” this past May in TechCrunch, Sarah Lacy portrayed a number of the hackers she had met in Lagos. In other circumstances, she said, the best of them might have been like Sergey Brin or Max Levchin, the immigrants who co-founded Google and PayPal, respectively. They were that clever and technically gifted. Or, more modestly, they could have been like the engineers and managers I’ve met over the years at Google, Microsoft, Intel, and other companies, at least half of whom were born and raised overseas. But these hacking entrepreneurs couldn’t get out of Nigeria, and so they dealt with the outside world via “Mugged in Madrid” messages. Shreyas Doshi, a Google senior project manager, said that the company had run analyses to see how much money the scams might produce. “With a variety of assumptions, we believe they could easily make about $500 a day, if not more,” he said; that many people fall for these scams.

The greatest practical fear for my wife and me was that, even if she eventually managed to retrieve her records, so much of our personal and financial data would be in someone else’s presumably hostile hands that we would spend our remaining years looking over our shoulders, wondering how and when something would be put to damaging use. At some point over the past six years, our correspondence would certainly have included every number or code that was important to us—credit-card numbers, bank-account information, medical info, and any other sensitive data you can imagine.

The long siege of identity theft could still happen, and I have put all the credit-card companies on alert. But a few months into our post-hacking life, we’ve seen no indication of trouble, and according to the Google officials, the hacker would depart from past patterns if he began using the data in the way we most feared. “We haven’t had a major incident with credit-card [numbers obtained from scanning Gmail] yet,” Bryant Gehring said. The economics of hacking are constantly changing, so we are still on guard. But security officials at Google and elsewhere said that for talented hackers in Lagos or Abidjan, it is generally safer and more rewarding to collect money at Western Union, which can be done virtually untraceably, as opposed to poring through e‑mail archives in search of data to create identity-theft schemes. Why, then, did the hacker bother to erase all the archives? “Because he—or she—was a good hacker,” Gehring said. “He wanted to make it harder for you to get in touch with all your contacts and warn them not to send money to Madrid. You remember their names but probably not all their e‑mail addresses.” Many e-mail hackings, whether on Gmail or another major system, involve deletion of mail that arrives after the hijacking; only a destructive minority involve a complete zeroing-out of the archives like what happened to my wife.

Against this assault, the Google security team, like its counterparts at other companies, is constantly monitoring activity across its systems, toward the end of detecting break-ins and hijacks before damage has been done, and even before the owners know that something has gone wrong. The signs they’re looking for have endlessly evolved: this year’s “Mugged in Madrid” message was last year’s “Mugged in London,” with enough minor changes to avoid many filters. “Google is a very data-centric place,” Eric Grosse, Google’s head of security, said to me. “We log lots of things and constantly have computers crunching over this data, trying to look for subtle patterns and keep up with the continual changes. That lets us do a better job of spam filtering than you could do on your own machine, since we see so much of it. We try to use the same techniques to look for other signs of abuse.” Jay Nancarrow, of Google’s communications team, added: “A lot of the signals we analyze are evident only because we’re offering the service to so many people. This is the kind of machine-learning the cloud enables that is very hard otherwise.”

I kept asking for examples of such tells; perhaps reassuringly, the Google officials kept saying they would rather not get into details. “But suppose we see that all your friends start marking mail from your account as spam,” Shreyas Doshi said. Obviously he didn’t mean that any human being at Google is monitoring your correspondence. Rather, its systems, like those of other companies that handle large numbers of online accounts, are constantly searching for anomalies. “Or if suddenly the recovery phone number in your account is changed to a number in South Africa. Or if you suddenly delete 20,000 e‑mails at once.” Any of these would be a sign that something might be going wrong, which Google would use to automatically immobilize the account. Indeed, the deletion of tens of thousands of e‑mails at once is what happened to my wife—and according to Gehring, the monitoring system had already noted an anomaly and was beginning to freeze the account by the time we filed our first report.

Perhaps the most startling thing I learned at Google about my wife’s case was how “lucky” we had been. Lucky not in having friends we could turn to in the otherwise automated and unapproachable Google edifice—though, of course, we were—but simply in the timing of the attack. If this had happened six months or a year earlier, or if it happened even today at most other e-mail services, the archives would likely have been gone forever. It was only because of the Undeletion Project that recovery, although slow, was feasible at all.

How could this be? How could big tech companies offer cloud services to hundreds of millions of people without better guarding their data against catastrophic loss? On Google’s side, one explanation involved complexities of the law. My wife and I might think that Google had a “duty” to be able to find her messages after some hacker had erased them. But according to Google’s legal department, its higher and more stringent duty is to ensure that messages are erased, if whoever is in charge of an account wants them gone. Political activists in repressive countries, people who for whatever reason (@RepWeiner) want parts of their electronic correspondence to disappear—they are the ones Google, like other e‑mail providers, had in mind in designing a system optimized for deletion rather than recovery. In exceptional cases, mainly in response to government orders in criminal or anti-terrorism investigations, Google could laboriously piece together already deleted records from its tape backups. But such recovery efforts were slow, hand-crafted undertakings, impractical for responding to thousands of hacking episodes a day (only some of which lead to total erasures).

“Every year, we do an immersion session with people involved in consumer operations,” I was told by Birendro Roy, one of the two young engineers at Google who developed the software for the Undeletion Project. He and his colleague Jaishankar Sundararaman were introduced to me as “the people who actually restored your wife’s mail.” (Roy, whose father is from India and mother is from Finland, grew up in Baton Rouge and went to MIT. Sundararaman grew up and went to college in Tamil Nadu, in southern India, and then did graduate work at Virginia Tech. I mention this as part of my ongoing chronicle of America’s strength through attracting foreign talent.) “The consumer reps tell us what are the common user ‘pain points,’ and restoring mail had definitely been something users were requesting.”

What a surprise, that people would want to recover from catastrophe! But from Google’s engineering perspective, the deleted-mail problem, while dire for those confronting it, affected only a tiny fraction of their users, and also was more complicated to solve than some other mainstream usability issues. In companies like Google, relatively few innovations are the result of top-down orders from executives. More evolve bottom-up, as engineers and product managers become sold on the need to add a new product or feature to the company’s offerings. Sometime last year, the Gmail engineers became sold on the value of a recovery system for maliciously deleted e‑mail. My wife’s luck was that her hack occurred this year rather than last—“Before, users were pretty much out of luck,” Bryant Gehring said—and that it hit the debugging cycle just when it did. To streamline a long story, the Undeletion Project software still had an important bug at the time of our first complaint. When the recovery software reached a message of a certain sort (“it was a ‘corner case’ involving saved drafts,” Sundararaman said), it mistakenly concluded that it had gotten to the end of the e‑mail list and could find nothing more. The engineers discovered the bug at just the time I was launching my “You cannot be serious!” screeds from outside. So now, if you lose your e-mail and start the recovery in time (deleted e-mail stays in a theoretically recoverable limbo for as long as 30 days, after which it’s likely gone for good), you should be able to get it back. In September, Google began offering a live help line for e-mail recovery, at a small cost (to deter abuse) of a few dollars.

“Of course, the very best way to recover data is not to lose it in the first place,” Shreyas Doshi said, in outlining what he described as a three-tier philosophy of data protection that was shared by Google, Yahoo, and other “cloud” companies. But that won’t help people like Deb when they make their horrifying discoveries. So Doshi outlined a tiered strategy I endorse for anyone who runs or uses e‑mail or Internet services—which is to say, everybody.

The first tier is for companies to give users a more realistic view of where and why they might be vulnerable, so they can do more to protect themselves. The second is for companies to reduce the incentive toward Internet crime by making it less profitable for the perpetrators—for instance, with better, faster ways of detecting hijackings, so that accounts will automatically be closed, even before the first “Mugged in Madrid” spam goes out. (And it seems that someone might usefully have a word with Western Union too.) The third is for companies to create better recovery systems, like the Undeletion Project, for when steps one and two fail.

Discussion of the cyber-threat menace has become a boom industry on its own. My latest immersion in the topic has left me more consoled than I might have expected—especially when I was imagining, early on, what the intrusion on my own household might mean to everyone else on the Internet who might innocently have exposed un‑backed‑up data to similar risk.

True, there is a lot to be worried about. “We have not examined a [corporate or governmental] network yet in which we have not discovered some level of intrusion,” Ron Ritchey, a longtime security expert now working at Booz Allen, told me. And “these things have been going on for years and years,” his colleague Ken Silva, the NSA veteran, said. “But companies are only now beginning to talk about them. After the Google report”—not about low-level cases like ours but about the massive Chinese intrusion on Google’s internal systems that the company revealed early last year—“the landscape for reporting these events really changed.” Both experts said the potential threat that concerned them most was not data theft or data loss, especially as it affected individual users, but data alteration that could affect institutions. “Imagine what could happen to the financial system, if trading data were altered,” Silva said. (Imagine: people might not trust financiers!) Or the medical data in a hospital. Or the formulas and data in spreadsheets on which companies base their bids and negotiations.

But on the other side, between us and data chaos are potentially protective factors of scale, technology, and strategy. Scale, in that big companies can invest big resources in monitoring and protecting their and their customers’ data. My money should be harder to steal if it’s in the bank than if it’s in my pocket, and something similar should be true of data. Technology, in that every person I spoke with emphasized that we’re living through an awkward stage in our current reliance on passwords. “Good passwords are bad for people, and bad passwords are good for criminals,” Michael Jones of Google said. Sooner or later, a better, easier, more reliable system of verifying identity will become widespread. It could be “biometric”—thumbprint recognition, iris scans, even a match of your face’s image, as you sit in front of the computer’s camera, with a stored picture of you. It could be purely digital, like Google’s recently introduced “two-step verification” system for Gmail identity, in which users enter both their password and a special code sent by Google to their cell phone or mobile device. None of these is perfect, but each one is a big advance over security now.

And strategy, in the form of a corporate and governmental emphasis on “data hardening” to limit the resulting loss when some attack inevitably succeeds. No free society can completely eliminate loss from crime, accidental death or destruction, even terrorism. The sane strategy is to prepare to contain the damage if and when things go wrong. Airbags in cars, watertight compartments in ships, hardened cockpit doors in airliners—these all illustrate the approach. The scary part in my wife’s case was that failure in one place meant potential failure of the system as a whole. Because she had used her password in one vulnerable place, all of the electronic data that mattered to her was at risk. If the data-hardening principle got through to me in the course of interviews, it has certainly occurred to security professionals at the institutions that collectively make up our new cloud info-system.

What about the rest of us, who are not security professionals? I asked that of every person I interviewed. Many of their recommendations boiled down to the hope that people would think more about their life online. “We’d like people to view their information life the way they view other parts of their life,” Andrew Kovacs of Google said. “It’s a good practice to review your financial situation every so often, and it’s a good practice to review your passwords and online-account information too.” Another official compared “cloud hygiene” to personal hygiene: you feel bad if you don’t brush your teeth or take a shower, and you should learn to feel bad if you’re taking risks online.

In practical terms, I have three action points I preach to anyone who will listen. Really, only two and a half.

The half step is extremely important, but I count it only as a half because it applies specifically to users of Gmail rather than other online systems. Here it is: if you use Gmail, please use Google’s new “two-step verification” system. In practice this means that to log into your account from any place other than your own computer, you have to enter an additional code, from Google, shown on your mobile phone. On your own computer, you enter a code only once every 30 days. This is not an airtight solution, but it can thwart nearly all of the remote attacks that affect Gmail thousands of times a day. Even though the hacker in Lagos has your password, if he doesn’t have your cell phone, he can’t get in.

In case you’ve missed the point: if you use Gmail, use this system. Also, make sure the recovery information for your account—a backup e-mail address or cell phone where you can receive password-reset information—is current. Google uses these to verify that you are the real owner.

Next we have password selection, that seemingly impossible task. The science, psychology, and sociology of creating strong passwords is a surprisingly well-chronicled and fascinating field. On The Atlantic’s Web site, we will describe some of the main strategies and the reasoning behind them. Even security professionals recognize the contradiction: the stronger the password, the less likely you are to remember it. Thus the Post-it notes with passwords, on monitor screens or in desk drawers.

But there is a middle ground, of passwords strong enough to create problems for hackers and still simple enough to be manageable. There are more details on our site, but strategies include:

• Choose a long, familiar-to-you sequence of ordinary words, with spaces between them as in an ordinary sentence, which more and more sites now allow. “Lake Winnebago is deep and chilly,” for instance. Or “my favorite packer is not brett favre.” You could remember a phrase like that, but a hacker’s computer, which couldn’t tell spaces from characters, would see only one forbiddingly long password sequence.

• Choose a shorter sequence of words that are not “real” English words. I once lived in a Ghanaian village called Assin Fosu. I can remember its name easily, but it would be hard to guess. Even harder if I added numbers or characters.

• Choose a truly obscure, gibberish password—“V*!amYEg5M5!3R” is one I generated just now with the LastPass system, and you’re welcome to it—and then find a way to store it. Having it written down in your wallet is one, though the paper it’s on shouldn’t say “Passwords” at the top. The approach I prefer, and use for some passwords, is to entrust them to online managers like LastPass or RoboForm. Even if their corporate sites were hacked, that wouldn’t reveal all your passwords, since the programs work by storing part of the encoding information in the cloud and part on your own machine.

At a minimum, any step up from “password,” “123456,” or your own birthday is worthwhile.

Finally, use different passwords. Not hundreds of different ones, for the hundreds of different places that require logins of some kind. The guide should be: any site that matters needs its own password—one you don’t currently use for any other site, and that you have never used anywhere else.

“Using an important password anywhere else is just like mailing your house key to anyone who might be making a delivery,” Michael Jones of Google said. “If you use your password in two places, it is not a valid password.”

I asked my experts how many passwords they personally used. The highest I heard was “about a dozen.” The lowest was four, and the norm was five or six. They all stressed that they managed their passwords and sites in different categories. In my own case, there are five sites whose security really matters to me: my main e‑mail account, two credit-card sites, a banking account, and an investment firm. Each has its own, good password, never used anywhere else. Next are the sites I’d just as soon not have compromised: airline-mileage accounts, Amazon and Barnes & Noble, various message boards and memberships. I have two or three semi-strong passwords I use among all of them. If you hacked one of them you might hack the others, but I don’t really care. Then there is everything else, the thicket of annoying little logins we all deal with. I have one or two passwords for them too. By making it easy to deal with unimportant accounts, I can concentrate on protecting the ones that matter.

At the end of my visit to Google, I went by to see Michael Jones, the friend to whom I’d first turned in data-loss despair. I told him what I’d learned, and how I would try to spread the message of shared responsibility, individual and organizational, for security in the cloud age.

“I see that you’ve got it!” he said. “The zeal of the convert. People in the business think about the risks all the time, but normal people don’t, until they’ve gotten a taste of the consequences of failure.”

I have now had that taste and am here to share the experience. As with so many other challenges in modern life, responding with panic or zealotry doesn’t get us anywhere. But a few simple self-protective steps can save a lot of heartache later on.

James Fallows is an Atlantic national correspondent; his blog is at jamesfallows.theatlantic.com.
Presented by

James Fallows is a national correspondent for The Atlantic and has written for the magazine since the late 1970s. He has reported extensively from outside the United States and once worked as President Carter's chief speechwriter. His latest book is China Airborne. More

James Fallows is based in Washington as a national correspondent for The Atlantic. He has worked for the magazine for nearly 30 years and in that time has also lived in Seattle, Berkeley, Austin, Tokyo, Kuala Lumpur, Shanghai, and Beijing. He was raised in Redlands, California, received his undergraduate degree in American history and literature from Harvard, and received a graduate degree in economics from Oxford as a Rhodes scholar. In addition to working for The Atlantic, he has spent two years as chief White House speechwriter for Jimmy Carter, two years as the editor of US News & World Report, and six months as a program designer at Microsoft. He is an instrument-rated private pilot. He is also now the chair in U.S. media at the U.S. Studies Centre at the University of Sydney, in Australia.

Fallows has been a finalist for the National Magazine Award five times and has won once; he has also won the American Book Award for nonfiction and a N.Y. Emmy award for the documentary series Doing Business in China. He was the founding chairman of the New America Foundation. His recent books Blind Into Baghdad (2006) and Postcards From Tomorrow Square (2009) are based on his writings for The Atlantic. His latest book is China Airborne. He is married to Deborah Fallows, author of the recent book Dreaming in Chinese. They have two married sons.

Fallows welcomes and frequently quotes from reader mail sent via the "Email" button below. Unless you specify otherwise, we consider any incoming mail available for possible quotation -- but not with the sender's real name unless you explicitly state that it may be used. If you are wondering why Fallows does not use a "Comments" field below his posts, please see previous explanations here and here.
Get Today's Top Stories in Your Inbox (preview)

The Time JFK Called the Air Force to Complain About a 'Silly Bastard'

51 years ago, President John F. Kennedy made a very angry phone call.


Elsewhere on the web

Video

Adventures in Legal Weed

Colorado is now well into its first year as the first state to legalize recreational marijuana. How's it going? James Hamblin visits Aspen.

Video

What Makes a Story Great?

The storytellers behind House of CardsandThis American Life reflect on the creative process.

Video

Tracing Sriracha's Origin to Thailand

Ever wonder how the wildly popular hot sauce got its name? It all started in Si Racha.

Video

Where Confiscated Wildlife Ends Up

A government facility outside of Denver houses more than a million products of the illegal wildlife trade, from tigers and bears to bald eagles.

Video

Is Wine Healthy?

James Hamblin prepares to impress his date with knowledge about the health benefits of wine.

Video

The World's Largest Balloon Festival

Nine days, more than 700 balloons, and a whole lot of hot air

Writers

Up
Down

More in Technology

More back issues, Sept 1995 to present.

Just In