How a dream team of engineers from Facebook, Twitter, and Google built the software that drove Barack Obama's reelection
The Obama campaign's technologists were tense and tired. It was game day and everything was going wrong.
Josh Thayer, the lead engineer of Narwhal, had just been informed that they'd lost another one of the services powering their software. That was bad: Narwhal was the code name for the data platform that underpinned the campaign and let it track voters and volunteers. If it broke, so would everything else.
They were talking with people at Amazon Web Services, but all they knew was that they had packet loss. Earlier that day, they lost their databases, their East Coast servers, and their memcache clusters. Thayer was ready to kill Nick Hatch, a DevOps engineer who was the official bearer of bad news. Another of their vendors, PalominoDB, was fixing databases, but needed to rebuild the replicas. It was going to take time, Hatch said. They didn't have time.
They'd been working 14-hour days, six or seven days a week, trying to reelect the president, and now everything had been broken at just the wrong time. It was like someone had written a Murphy's Law algorithm and deployed it at scale.
And that was the point. "Game day" was October 21. The election was still 17 days away, and this was a live action role playing (LARPing!) exercise that the campaign's chief technology officer, Harper Reed, was inflicting on his team. "We worked through every possible disaster situation," Reed said. "We did three actual all-day sessions of destroying everything we had built."
Hatch was playing the role of dungeon master, calling out devilishly complex scenarios that were designed to test each and every piece of their system as they entered the exponential traffic-growth phase of the election. Mark Trammell, an engineer who Reed hired after he left Twitter, saw a couple game days. He said they reminded him of his time in the Navy. "You ran firefighting drills over and over and over, to make sure that you not just know what you're doing," he said, "but you're calm because you know you can handle your shit."
The team had elite and, for tech, senior talent -- by which I mean that most of them were in their 30s -- from Twitter, Google, Facebook, Craigslist, Quora, and some of Chicago's own software companies such as Orbitz and Threadless, where Reed had been CTO. But even these people, maybe *especially* these people, knew enough about technology not to trust it. "I think the Republicans fucked up in the hubris department," Reed told me. "I know we had the best technology team I've ever worked with, but we didn't know if it would work. I was incredibly confident it would work. I was betting a lot on it. We had time. We had resources. We had done what we thought would work, and it still could have broken. Something could have happened."
In fact, the day after the October 21 game day, Amazon services -- on which the whole campaign's tech presence was built -- went down. "We didn't have any downtime because we had done that scenario already," Reed said. Hurricane Sandy hit on another game day, October 29, threatening the campaign's whole East Coast infrastructure. "We created a hot backup of all our applications to US-west in preparation for US-east to go down hard," Reed said.
"We knew what to do," Reed maintained, no matter what the scenario was. "We had a runbook that said if this happens, you do this, this, and this. They did not do that with Orca."
THE NEW CHICAGO MACHINE vs. THE GRAND OLD PARTY
Orca was supposed to be the Republican answer to Obama's perceived tech advantage. In the days leading up to the election, the Romney campaign pushed its
(not-so) secret weapon as the answer to the Democrats' vaunted ground game. Orca was going to allow volunteers at polling places to update the Romney
camp's database of voters in real time as people cast their ballots. That would supposedly allow them to deploy resources more efficiently and wring every
last vote out of Florida, Ohio, and the other battleground states. The product got its name,
a Romney spokesperson told
, because orcas are the only known predator of the one-tusked narwhal.
The billing the Republicans gave the tool confused almost everyone inside the Obama campaign. Narwhal wasn't an app for a smartphone. It was the architecture of the company's sophisticated data operation. Narwhal unified what Obama for America knew about voters, canvassers, event-goers, and phone-bankers, and it did it in real time. From the descriptions of the Romney camp's software that were available then and now, Orca was not even in the same category as Narwhal. It was like touting the iPad as a Facebook killer, or comparing a GPS device to an engine. And besides, in the scheme of a campaign, a digitized strike list is cool, but it's not, like, a gamechanger. It's just a nice thing to have.
So, it was with more than a hint of schadenfreude that Reed's team hear that Orca crashed early on election day. Later reports posted by rank-and-file volunteers describe chaos descending on the polling locations as only a fraction of the tens of thousands of volunteers organized for the effort were able to use it properly to turn out the vote.
Of course, they couldn't snicker too loudly. Obama's campaign had created a similar app in 2008 called Houdini. As detailed in Sasha Issenberg's groundbreaking book, The Victory Lab, Houdini's rollout went great until about 9:30am Eastern on the day of the election. Then it crashed in much the same way Orca did.
In 2012, Democrats had a new version, built by the vendor, NGP VAN. It was called Gordon, after the man who killed Houdini. But the 2008 failure, among other needs, drove the 2012 Obama team to bring technologists in-house.
With election day bearing down on them, they knew they could not go down. And yet they had to accommodate much more strain on the systems as interest in the election picked up toward the end, as it always does. Mark Trammell, who worked for Twitter during its period of exponential growth, thought it would have been easy for the Obama team to fall into many of the pitfalls that the social network did back then. But while the problems of scaling both technology and culture quickly might have been similar, the stakes were much higher. A fail whale (cough) in the days leading up to or on November 6 would have been neither charming nor funny. In a race that at least some people thought might be very close, it could have cost the President the election.
And of course, the team's only real goal was to elect the President. "We have to elect the President. We don't need to sell our software to Oracle," Reed told his team. But the secondary impact of their success or failure would be to prove that campaigns could effectively hire and deploy top-level programming talent. If they failed, it would be evidence that this stuff might be best left to outside political technology consultants, by whom the arena had long been handled. If Reed's team succeeded, engineers might become as enshrined in the mechanics of campaigns as social-media teams already are.
We now know what happened. The grand technology experiment worked. So little went wrong that Trammell and Reed even had time to cook up a little pin to celebrate. It said, "YOLO," short for "You Only Live Once," with the Obama Os.
When Obama campaign chief Jim Messina signed off on hiring Reed, he told him, "Welcome to the team. Don't fuck it up." As Election Day ended and the dust settled, it was clear: Reed had not fucked it up.
The campaign had turned out more volunteers and gotten more donors than in 2008. Sure, the field organization was more entrenched and experienced, but the difference stemmed in large part from better technology. The tech team's key products -- Dashboard, the Call Tool, the Facebook Blaster, the PeopleMatcher, and Narwhal -- made it simpler and easier for anyone to engage with the President's reelection effort.
But it wasn't easy. Reed's team came in as outsiders to the campaign and by most accounts, remained that way. The divisions among the tech, digital, and analytics team never quite got resolved, even if the end product has salved the sore spots that developed over the stressful months. At their worst, in early 2012, the cultural differences between tech and everybody else threatened to derail the whole grand experiment.
By the end, the campaign produced exactly what it should have: a hybrid of the desires of everyone on Obama's team. They raised hundreds of millions of dollars online, made unprecedented progress in voter targeting, and built everything atop the most stable technical infrastructure of any presidential campaign. To go a step further, I'd even say that this clash of cultures was a good thing: The nerds shook up an ossifying Democratic tech structure and the politicos taught the nerds a thing or two about stress, small-p politics, and the significance of elections.
YOLO: MEET THE OBAMA CAMPAIGN'S CHIEF TECHNOLOGY OFFICER
If you're a nerd, Harper Reed is an easy guy to like. He's brash and funny and smart. He gets you and where you came from. He, too, played with computers when they weren't cool, and learned to code because he just could not help himself. You could call out nouns, phenomena, and he'd be right there with you: BBS, warez, self-organizing systems, Rails, the quantified self, Singularity. He wrote his first programs at age seven, games that his mom typed into their Apple IIC. He, too, has a memory that all nerds share: Late at night, light from a chunky monitor illuminating his face, fingers flying across a keyboard, he figured something out.
TV news segments about cybersecurity might look lifted straight from his memories, but the b-roll they shot of darkened rooms and typing hands could not convey the sense of exhilaration he felt when he built something that works. Harper Reed got the city of Chicago to create an open and real-time feed of its transit data by reverse engineering how they served bus location information. Why? Because it made his wife Hiromi's commute a little easier. Because it was fun to extract the data from the bureaucracy and make it available to anyone who wanted it. Because he is a nerd.
Yet Reed has friends like the manager of the hip-hop club Empire who, when we walk into the place early on the Friday after the election, says, "Let me grab you a shot." Surprisingly, Harper Reed is a chilled vodka kind of guy. Unsurprisingly, Harper Reed read Steven Levy's Hackers as a kid. Surprisingly, the manager, who is tall and handsome with rock-and-roll hair flowing from beneath a red beanie, returns to show Harper photographs of his kids. They've known each other for a long while. They are really growing up.
As the night rolls on, and the club starts to fill up, another friend approached us: DJ Hiroki, who was spinning that night. Harper Reed knows the DJ. Of course. And Hiroki grabs us another shot. (At this point I'm thinking, "By the end of the night, either I pass out or Reed tells me something good.") Hiroki's been DJing at Empire for years, since Harper Reed was the crazy guy you can see on his public Facebook photos. In one shot from 2006, a skinny Reed sits in a bathtub with a beer in his hand, two thick band tattoos running across his chest and shoulders. He is not wearing any clothes. The caption reads, "Stop staring, it's not there i swear!" What makes Harper Reed different isn't just that the photo exists, but that he kept it public during the election.
Yet if you've spent a lot of time around tech people, around Burning Man devotees, around startups, around San Francisco, around BBSs, around Reddit, Harper Reed probably makes sense to you. He's a cool hacker. He gets profiled by Mother Jones even though he couldn't talk with Tim Murphy, their reporter. He supports open source. He likes Japan. He says fuck a lot. He goes to hipster bars that serve vegan Mexican food, and where a quarter of the staff and clientele have mustaches.
He may be like you, but he also juggles better than you, and is wilder than you, more fun than you, cooler than you. He's what a king of the nerds really looks like. Sure, he might grow a beard and put on a little potbelly, but he wouldn't tuck in his t-shirt. He is not that kind of nerd. Instead, he's got plugs in his ears and a shock of gloriously product-mussed hair and hipster glasses and he doesn't own a long-sleeve dress shirt, in case you were wondering.
"Harper is an easy guy to underestimate because he looks funny. That might be part of his brand," said Chris Sacca, a well-known Silicon Valley venture capitalist and major Obama bundler who brought a team of more than a dozen technologists out for an Obama campaign hack day.
Reed, for his part, has the kind of self-awareness that faces outward. His self-announced flaws bristle like quills. "I always look like a fucking idiot," Reed told me. "And if you look like an asshole, you have to be really good."
It was a lesson he learned early out in Greeley, Colorado, where he grew up. "I had this experience where my dad hired someone to help him out because his network was messed up and he wanted me to watch. And this was at a very unfortunate time in my life where I was wearing very baggy pants and I had a Marilyn Manson shirt on and I looked like an asshole. And my father took me aside and was like, 'Why do you look like an asshole?' And I was like, 'I don't know. I don't have an answer.' But I realized I was just as good as the guys fixing it," Reed recalled. "And they didn't look like me and I didn't look like them. And if I'm going to do this, and look like an idiot, I have to step up. Like if we're all at zero, I have to be at 10 because I have this stupid mustache."
And in fact, he may actually be at 10. Sacca said that with technical people, it's one thing to look at their resumes and another to see how they are viewed among their peers. "And it was amazing how many incredibly well regarded hackers that I follow on Twitter rejoiced and celebrated [when Reed was hired]," Sacca said. "Lots of guys who know how to spit out code, they really bought that."
By the time Sacca brought his Silicon Valley contingent out to Chicago, he called the technical team "top notch." After all, we're talking about a group of people who had Eric Schmidt sitting in with them on Election Day. You had to be good. The tech world was watching.
Terry Howerton, the head of the Illinois Technology Association and a frank observer of Chicago's tech scene, had only glowing things to say about Reed. "Harper Reed? I think he's wicked smart," Howerton said. "He knows how to pull people together. I think that was probably what attracted the rest of the people there. Harper is responsible for a huge percentage of the people who went over there."
Reed's own team found their co-workers particularly impressive. One testament to that is several startups might spin out of the connections people made at the OFA headquarters, such as Optimizely, a tool for website A/B testing, which spun out of Obama's 2008 bid. (Sacca's actually an investor in that one, too.)
"A CTO role is a weird thing," said Carol Davidsen, who left Microsoft to become the product manager for Narwhal. "The primary responsibility is getting good engineers. And there really was no one else like him that could have assembled these people that quickly and get them to take a pay cut and move to Chicago."
And yet, the very things that make Reed an interesting and beloved person are the same things that make him an unlikely pick to become the chief technology officer of the reelection campaign of the President of the United States. Political people wear khakis. They only own long-sleeve dress shirts. Their old photos on Facebook show them canvassing for local politicians and winning cross-country meets.
I asked Michael Slaby, Obama's 2008 chief technology officer, and the guy who hired Harper Reed this time around, if it wasn't risky to hire this wild guy into a presidential campaign. "It's funny to hear you call it risky, it seems obvious to me," Slaby said. "It seems crazy to hire someone like me as CTO when you could have someone like Harper as CTO."