Letter: Is ‘The Geography of Partisan Prejudice’ Knowable?

A reader questions the methodology behind The Atlantic’s guide to the most—and least—politically open-minded counties in America.

The Geography of Partisan Prejudice

In early March, The Atlantic published a guide to the most—and least—politically open-minded counties in America. Amanda Ripley, Rekha Tenjarla, and Angela Y. He teamed up with PredictWise, a polling and analytics firm, to create a ranking of counties in the U.S. based on partisan prejudice (or what researchers call “affective polarization”).

The results were surprising in several ways, they found. “In general, the most politically intolerant Americans, according to the analysis, tend to be whiter, more highly educated, older, more urban, and more partisan themselves.”

---

In its analysis, PredictWise used multilevel regression with post-stratification (MRP), a method that has lots of promise in estimating public opinion within states from national survey data. However, this method should be used with great care and attention to detail. In particular, the method has been shown to produce estimates that are highly variable when sample sizes are small, as demonstrated in this 2017 paper by Matthew K. Buttice and Benjamin Highton.

In my view, a substantial issue with the methodology is that the sample size used in this study (N=2,000) indeed seems too small to detect real differences in partisan prejudice between states, much less differences between counties. This might explain why levels of partisan prejudice appear to differ so greatly between counties in North Carolina and South Carolina. It seems highly unlikely that partisan prejudices respect the Carolina border in such a way, yet that is exactly what the results suggest. Consider further that if states are sampled in proportion to population, something like 60 North Carolinians were likely included in this sample. That doesn’t seem nearly enough to make any strong claims about the state as a whole, much less the 100 counties the state is composed of, without making some grand statistical assumptions. Even with a method such as MRP, it’s unlikely there’s enough coverage here to generalize findings to all 3,007 U.S. counties. I’d be interested to see confidence intervals around the estimates of partisan prejudice for both states and counties—I suspect they are quite wide. The data produced here are at best interesting “guesses” derived from small data, and the level of confidence in these results seems largely overstated.

Another thought to consider, mentioned only briefly in the article, is that these results might also be reflective of who is currently in the White House. Educated, white liberals might feel more hostility than usual toward conservatives given Donald Trump’s election, while conservatives might feel more motivated to prove that they are reasonable and fair voters. I wonder whether the findings might have been totally different in 2012 or would be entirely different if this was conducted again in 2021. Certainly, self-reported survey answers should always be interpreted in the context of the environment and time in which the survey was conducted.

Aside from my methodological concerns, I grew up in Davidson County, North Carolina, described in the article as being in the 4th percentile of partisan prejudice. I find this very difficult to believe. Many high-school friends decided to end our friendship when they learned I voted for Barack Obama, despite the fact that I did not end friendships when classmates attempted to convince me that then–Senator Obama was the anti-Christ (You can’t make this stuff up!).

In short, I fear The Atlantic has erred in the publishing of this article. I am sure that the general finding that partisan prejudice is stronger among white, educated liberals on average among U.S. citizens in 2019 is reasonable. But I’m very skeptical that this can be used to draw conclusions about differences in partisan prejudice among states and counties.

Scott James
B.S. Statistics, M.S. Advanced Analytics
Arlington, Va.

---

Tobias Konitzer, a co-founder of PredictWise, replies:

Thank you very much for your thoughtful critique. Let me address your methodological concerns first: The study attempts to shed light on a poorly understood phenomenon—political tolerance—and discover how it varies from place to place. We started by asking our respondents 14 survey questions, such as how selfish and patriotic they would rate members of each party and how upset people would be if their offspring married a Democrat or a Republican (full survey here).

We then identified how individual-level demographics (such as education, age, and race) and the mix of our respondents’ neighborhoods relate to political tolerance. In this way, we created neighborhood “profiles” of political tolerance. Then we used a statistical trick: projecting this profile onto a massive data set including roughly 250,000,000 Americans. This large baseline data set, built using commercial voter files, includes millions of combinations of factors and has been carefully curated by PredictWise over the past four years.

This trick prevents us from having to collect survey data in every county in the U.S.—something which would cost well north of $15 million and is therefore unfeasible for news outlets. As you point out, this method (which we call Mr. P) has shown promise in estimating political opinion in smaller geographic areas. In fact, my co-founder David Rothschild and I have extensively published around the methodology in leading academic journals (here, here, and here).

In contrast to what you suggest in your note, however, our model does not attempt to identify state-level dynamics with an insufficient sample size. Instead, our assumption is that counties composed of residents with similar demographic and neighborhood profiles will perform similarly on our political-tolerance scale. In essence, we overcome the traditional limitations of small sample sizes by relying on our massive baseline dataset to identify the range of different neighborhood profiles within and across counties.

Our goal in this undertaking was to combine quantitative methods with qualitative reporting. Ultimately, we tried to find both quantitative and qualitative evidence in support of positions toward the top or bottom of our tolerance scale (examples here).

Yes, there are caveats to this analysis:

1. Our methods have limits, the most important of which we believe to be the difficulty of counting political partisans at the county level. For example, we want to know how many white, married Republicans over age 55 with a college education happen to live in suburban neighborhoods with a high mix of partisans and different age groups in, say, Wake County, North Carolina. No one knows this exact number. Our large-scale data set on all adult Americans attempts to estimate this number, but it can introduce bias due to the slightly different ways Republicans and Democrats are counted across states (more here).
2. Our analysis is based on 2018 data, and we absolutely believe results need to be seen within the context of that time. The same is true for any poll or study on dynamic political attitudes.
3. We might miss county-level peculiarities beyond demographics and composition of neighborhoods, although we do think the data on neighborhood compositions offers a pretty good stand-in for such peculiarities. For example, Americans who live in politically segregated neighborhoods—something our model picks up—are less likely to have politically cross-cutting conversations at the grocery store than Americans who live in politically diverse neighborhoods.