Do We Want to Believe the Numbers? A Q&A With Nate Silver
The FiveThirtyEight founder on broken-windows policing, Max Scherzer, and foxes in the Oval Office.
Nate Silver’s book, The Signal and the Noise, is about making predictions. It examines what leads people to get predictions wrong, and what leads people to get them right—like Silver did when his model correctly predicted the result of the 2012 presidential election in all 50 states. That triumph led Jon Stewart to crown Silver “the lord and God of the algorithm.” Silver has since taken his approach to statistical analysis to FiveThirtyEight, where he is the editor-in-chief.
Ahead of the release this week of the paperback edition of his book, I spoke to Silver about sports, politics, and the theories we shouldn’t trust.
Noah Gordon: You’ve written a new introduction for the paperback edition of The Signal and the Noise. Have statistical predictions improved much since 2012, when many analysts overrated Romney’s chances? Do people put too much faith in them?
Nate Silver: I’m not sure I’d say people put too much faith in statistics or analytics. On the whole I’d say people put too little. But I think there’s a gap between the increasing desire to use analytics and where the craft is in terms of the practice. One irony about 2014 is that you actually had Democrats making a lot of the same mistakes that Republicans did in 2012. In defiance of where the polls stood they thought, oh, Democrats are going to keep the Senate. They came up with reasons why, like “our turnout operation will save us,” or “the demographics are favorable for us,” or they’d cite 2012 as a precedent. Until it actually happened, and the election was kind of a disaster for Democrats. You know, they were making the same mistakes. Less high-profile mistakes because people aren’t as concerned about the midterms as the presidency, but still—the average poll wound up overrating how Democrats would do by about three or four percent…
Recommended Reading
Gordon: Do people make those mistakes because they don’t want to believe the numbers?
Silver: I think that’s a lot of it, yeah. Confirmation bias is a big part of the story here. Another big part of the story is looking too narrowly at only the most recent event. So they would say, look, in 2012 the polls had a bias against Democrats, but if you look at the longer term, then there’s not really any long-term bias. Polls can be off, but it’s really difficult to predict in which direction they’ll be off ahead of time, and attempts to do it have proven to be a little bit foolish. In elections and politics you have events that occur only infrequently—elections occur every four years, or every two years I suppose. There’s not a lot of time to get the rapid feedback that you would in something like sports where you play a game every day, or other every day. So you can have your system be wrong for a long time. It means when you’re looking at historical data, you really have to use all the data you can instead of sort of asserting that well, we’re in a new paradigm now. Using a sample size of one or two is usually big mistake … What happened [in 2012] was not at all unprecedented if people take a broader rather than a narrower view of history.
Gordon: Let’s switch gears: [pitcher] Max Scherzer just left your hometown Tigers to sign a deal with my hometown Nationals. And the history of these big, long-term deals for pitchers is not good. Can Scherzer break the mold?
Silver: So I don’t know if have any comment about him in particular. I mean, there are a couple of things going on with these very substantial contracts in baseball. One being that teams are assuming that the inflation we’ve had in player salaries for many years will continue. You know, baseball salaries and [those of] other major sports, have grown at a rate much faster than inflation. If you’re signing a deal that goes out five or 10 years in the future, then you’re making a big bet on inflation … I think most teams have internalized the lesson in the abstract that it’s a bad idea to sign a pitcher to a long-term contract, and go back at look at multi-year contracts, especially for pitchers, turn out badly four times out of five. And for hitters, seven times out of 10.
You have what’s sometimes called the winner’s curse in a market like this. Whichever team defies the consensus and says “we think this [guy] is even better than everyone else does, we’re going to pay him more”… If you look at The Signal and the Noise, one of the themes is that, in general, consensus predictions are more accurate than predictions based on what any one person might say. So here, whoever’s most optimistic on Scherzer is going to get him because they’ll offer the most salary. But they’re probably overpaying based on where the market should be. You can’t short a baseball players contract like you would a stock, so there’s no mechanism determining what a baseball player’s salary should be.
Gordon: Yeah, inflation can make a huge difference. I guess we’ll have to hope that Janet Yellen becomes a Nationals fan.
Silver: [laughs] In the NBA, where they’re expecting this rather substantial increase in the salary cap, some deals that seemed bad a couple of years ago seem more defensible if you have 10 percent growth in the salary cap the next couple of seasons. But I don’t think teams are signing those contracts for those reasons … There’s a chance we’re in a little bit of a bubble with sports team valuations. If you look what the Clippers were sold for …
Gordon: That [$2 billion] deal was insane.
Silver: Yeah! Part of it ties back to global economics, but part of it has to do with the rise of these super wealthy multi-millionaires and billionaires, really. If a franchise literally costs a billion dollars, give or take, are there enough billionaires in the world to potentially purchase an asset like that? There are. That number has been increasing steadily of the past couple decades. The market for sports franchises, the market for fine art, they have really been booming. In some ways, you are kind of making a bet on the 0.1 percent—that there will be more and more members of that class. That you can sell to more potential buyers. If that turns out not to be true, you could have a crash in the baseball market or the art market or whatever else.
Gordon: Do you think basketball is experiencing its own sort of Moneyball revolution, as more data and metrics become available?
Silver: The basketball stuff is pretty darn impressive. You have an ex-MIT guy, Daryl Morey, running one of the best franchises in the league in Houston. But people might not realize that basketball is almost as far along as baseball. And what’s funny about both those sports—really all sports—is you’re getting a lot of new data. Where now the NBA can actually track where is a player is on the court at any moment, can actually see who’s hustling and who’s not, what defensive positioning is like. What’s disappointing about the NBA relative to baseball is that much less data is public knowledge. In baseball, where stat geeks were outsiders for so long, you had a tradition of research that was made public, and free, and open source for academics, so people could get the basics of it. I’m sure teams are doing even more sophisticated stuff [internally] now. But in the NBA teams will gobble up analytic talent really quite quickly, and of course when teams are doing things that give them some type of advantage, that are proprietary, they don’t really want to share it ...
Gordon: Let's go back to the book. You put Malcolm Gladwell in the hedgehog category because of his belief in the "tipping point" theory. Does that theory hold up? [“Hedgehog” comes from Isaiah Berlin's categorization of thinkers into two broad categories: hedgehogs, who understand the world through the lens of a single defining idea; and foxes, whose ideas and experiences are more diffuse.]
Silver: I think the notion that processes involve feedback loops is correct. The growth in baseball salaries can be a feedback loop of sorts. That’s broadly true. But I’m suspicious of theories that are meant to explain everything. You know, the world is a complicated place. Human beings are smart, but we’re not all that bright compared to how complex the universe we’re trying to analyze is. So I’m suspicious of kind of putting a big tagline on something and saying it explains every phenomenon you might want. I think Malcolm Gladwell is a tremendous communicator of ideas and provocateur, but it’s a simplification oftentimes of what the story might be.
Gordon: Is Barack Obama a hedgehog or a fox?
Silver: I think Barack Obama is a fox, and in some ways that’s part of what’s made his presidency difficult. Because George W. Bush was the ultimate hedgehog, really. He had a very strong sense and would communicate as such to that public that here’s what the moral order of the universe is; things are absolutely good or evil. There’s not a lot of ambiguity really. Hedgehogs have an opportunity to be really decisive sometimes, and persuasive, and they can also be totally wrong about something, and you know, start a war over it.
Whereas foxes are more deliberative a lot of the time. And that can be frustrating to people when they want a quick response. At the same time you could say Obama’s playing a long game. You look at historians, they say he might look better in 10 years than he does now. His approval ratings are up a little bit. But a lot of times people want Obama to respond more strongly and firmly to things—that’s not always in the fox’s DNA. They want to collect more data, get more evidence, they’re more comfortable with ambiguity and uncertainty.
The evidence says that being a fox is a good thing if you’re trying to make forecasts or predictions about the world. It doesn’t say it’s the only personality trait that you’d want for success in any line of work. I would think most CEOs are hedgehogs, they believe very strongly in one particular vision of their product. They’re probably overly optimistic about how good their product will do. Many of them fail.
Gordon: But sometimes you get the iPhone.
Silver: Yeah, sometimes you get the iPhone, but sometimes you get all the imitations of the iPhone that have failed. Whereas foxes are usually a little better about hedging their bets and managing risks. So it’s sometimes a question of which personality traits you want in a CEO or a president … Bush was kind of a hedgehog and people have a long history of overcompensating for the previous president when they pick the next one. I think you’ll see themes in 2016 where a president who seems more decisive, that might be a persuasive characteristic for people.
Gordon: In your book you also write about magic bullet theories, for example, [economist] Douglas Hibbs’s theory that you could predict an election by looking only at economic growth and military casualties. Which magic bullet theories do people still put stock in today?
Silver: There are a lot of them, I suppose. This is kind of where the Gladwell comparison comes in. It’s also partly—you’ll sometimes take research that is nuanced and complex and it’ll be simplified in the media’s understanding of it. One that jumps out at me a lot is this so-called broken-windows theory, where the idea that by deterring small crimes, acts of vandalism, you create an environment where big crime is less likely to occur—that idea has been taken as true by a lot of police departments and there’s never been any particular empirical evidence behind it, or at least the evidence is rather mixed. I mean, that’s a theory that has had a lot of influence relative to how much empirical weight there really is behind it. Think about the incentives of running a police department. Well, it’s really hard to catch rapists and murderers and bank robbers. It’s not that hard to catch a graffiti artist or a small-time drug dealer, or some adolescents out where they shouldn’t be. So they can kind of score a lot of small wins by doing that. They’ll say arrests are up, we’re deterring more crime, we have a lot of data now on crime—but whether that translates into reductions in big-picture crime is not clear. And whether this leads to a more abusive relationship between the police department and the community—obviously that’s a big topic now—that’s not clear either …
We noticed in the neighborhood in Staten Island where Eric Garner wound up being killed, cops are running a ton of tickets in that neighborhood for minor offenses. We’re getting on a tangent here, but this [theory] is one that strikes me a big attractive-seeming theory that was very intriguing at the time, but people have sometimes stopped saying, “Hey, is this theory actually true?” when it’s convenient for them to believe in it.
Gordon: Say you could perfectly predict the outcome of an election, or a referendum about, say, gun control, what would it tell you about society? How can a good prediction shape society?
Silver: I’m kind of a purist in the sense that I don’t think the goal of a prediction is to shape society. To me the hypothetical where it’s like, “can you predict how an election will turn out exactly?” is like saying well, can you play God. It’s a very strange hypothetical. There are certainly cases where forecasts and predictions can inform policy-making. The irony is that sometimes those predictions become self-canceling. If the CDC says, "We have a potentially really bad flu outbreak this winter, so stay home from work and school, get your vaccinations or it’s going to be really bad," people actually for the most part take that advice. People take precautions and you have a rather mild year for the flu instead. Or if you look at directions your GPS might give you, there’s some evidence that in some cities, New York for example, the GPS will say FDR [bridge] is really jammed up, take the West Side highway instead. Every single driver now has a GPS unit telling him or her the same thing, so you now have the opposite problem. That’s why it’s tricky. When you’re making predictions in real time and it affects people’s behavior, it can be self-canceling or self-enhancing at different times.