The Existential Problem With the Polls

If Republicans opt out, polling as we know it will be beyond repair

(Photo by Brandon Bell/Getty Images)

I’ve always had an ambivalent feeling about polls. Individual polls mean little, since they are merely snapshots of a particular time period. Polls in the aggregate mean much more, and can tell us valuable information about public opinion or the state of a campaign. Over the last decade, with the rise of FiveThirtyEight and the New York Times’ Upshot, we’ve seen regular people become far more interested in and fluid in the lingo of pollsters. Many polls are quite good. Some are not. And many are taken out of context. There is the pitfall, which I’ve run into in my own reporting, of basing too much of what is written and said on what certain polls might be telling us. In a time when people are abandoning landlines, there is a risk that pollsters, unable to reach a representative sample, won’t give us accurate data.

It should be said, before I go on, that the pollsters who followed the New York City mayoral race performed, overall, fairly well. The polls accurately tracked the rise and fall of several candidates—most pollsters were able to account for Andrew Yang’s swift descent and Kathryn Garcia’s surge—and the first-place vote on election night offered few surprises in the Democratic primary. Eric Adams led, as all polls indicated, and Maya Wiley and Garcia were in contention. Some polls, later in the race, may have oversold Yang or Scott Stringer’s support, but this is no grave sin. Considering the traditional pollsters, for foolish reasons, sat out the mayoral race altogether, those who stepped up should be commended.

These polls were successful because Democrats, particularly in New York, have not given up on talking to pollsters. Certain demographics are harder to reach than others, but a pollster can always adjust for income, race, education, or age. The 2021 polls were, in the aggregate, as reliable as the polls in 2013. Few registered Democrats in the five boroughs, or in any city or state generally, will react with disdain if someone from Emerson, Data for Progress, Quinnipiac, NBC, or the New York Times calls or texts them.

Republicans are another matter entirely and I’m still not sure why this isn’t getting more attention. In 2016, there was a collective media meltdown after Donald Trump defeated Hillary Clinton, with many indicting the statewide and national polls for underestimating Trump’s support. Mea culpas abounded. Though people like Nate Silver were more willing than most to argue that Trump, once he seized the Republican nomination, had a chance of victory, pollsters and data analysts came in for a drubbing. Few offered viable alternative models for covering elections—Jay Rosen’s admirable “citizen’s agenda” being the exception—but the media and pundit class was sure, for the time being, something had to be done. The polls, they said, missed the mark.

The unsettling truth is that the much-maligned 2016 polls, which accurately forecasted Clinton’s popular vote victory and misfired in a few key swing states, performed far better than the 2020 polls. A new report from the American Association for Public Opinion Research confirmed this: 2020’s pre-election polls, both nationally and on the state-level, did not account for significant Republican support for Trump and GOP candidates down the ballot. While the 2016 fiasco was more a problem of interpretation than anything else—journalists and liberals could not fathom a world where Trump won, even though most polls forecasted a close enough popular vote—the 2020 failure was endemic to the polls themselves.

Final polls showed Biden crushing Trump in swing states like Wisconsin and Florida that Trump either would go on to win outright or narrowly lost. Republicans like Susan Collins coasted to victory in races where they had trailed throughout the summer and fall. A massive surge in turnout for Democrats did not lead to gains in the House. Republicans, rather, increased their numbers and threatened the Democratic majority.

“The 2020 polls featured polling error of an unusual magnitude: It was the highest in 40 years for the national popular vote and the highest in at least 20 years for state-level estimates of the vote in presidential, senatorial, and gubernatorial contests,” the AAPOR report read.

The report found that polling worsened in downballot contests. For senatorial and gubernatorial races combined, polls on average were six percentage points too favorable for Democratic candidates relative to the vote margin. Within the same state, the polling error was often larger in senatorial contests than the presidential race. On average, polls overstated the Democratic-Republican margin in states more supportive of Trump in 2016. Even after controlling for state-level differences in demographics and voting administration, the average error was larger in states that favored Trump in 2016.

None of the adjustments from 2016 worked. Weighting more for education, which was discussed so often in the run-up to 2020, did not matter. Late-deciding Republican voters did not cause the polls to misfire. And the faulty polls, according to the report, were not primarily caused by respondents’ reluctance to tell interviewers they supported Trump. In one sense, the “shy Trump voter” thesis was ruled out.

But another explanation looms, one that the report strongly considers: many Republicans are not responding to pollsters at all. Those that do, meanwhile, are more likely to support liberal-ish candidates anyway. “If the voters most supportive of Trump were least likely to participate in polls then the polling error may be explained as follows: Self-identified Republicans who choose to respond to polls are more likely to support Democrats and those who choose not to respond to polls are more likely to support Republicans,” the authors of the report explained. “Even if the correct percentage of self-identified Republicans were polled, differences in the Republicans who did and did not respond could produce the observed polling error.”

“This hypothesis is not unreasonable, considering the decreasing trust in institutions and polls especially among Republicans,” the report continued. “If so, self-identified Republican voters who participated in polls may have been more likely to support Democrats than those who chose not to participate in poll.”

Without knowing how respondents and nonrespondents compare—there is no way to evaluate how those who never answered the polls voted—there can only be so much data produced to undergird this thesis. But the report was able to determine that non-response did at least explain some of the errors in the polls across America.

“The overstatement of Democratic support could be attributed to unit nonresponse in several ways: between-party nonresponse, that is, too many Democrats and too few Republicans responding to the polls; within-party nonresponse, that is, differences in the Republicans and Democrats who did and did not respond to polls; or issues related to new voters and unaffiliated voters in terms of size (too many or too few) or representativeness (for example, were the new voters who responded to polls more likely to support Biden than new voters who did not respond to the polls?).”

The implications here are profound. National and local polls struggled to account for Republican support, most likely, because many Republicans could not be reached by phone or text. Before the rise of Trump, this was not an issue that manifested itself in data. A segment of the Republican base may have been consuming Fox News and resenting other media outlets and institutions, but they could still pick up the phone for a pollster. Trump was the first president to openly obsess over polls and declare open war with the media, rather than snipe at reporters in passing or fume about them in private, like Richard Nixon. Any media outlet Trump disliked, of course, was “fake news.” Any poll that showed him struggling or didn’t meet his approval was “fake.”

These days, even in defeat, Trump dominates the Republican Party to a remarkable degree. He is a pantheon figure for many Republicans, a new Ronald Reagan. If polls, in his view, are fake, then his voters may see them that way too. If someone calls from a pollster who belongs to one of the many organizations Trump has denigrated—CNN, NBC, CBS, the Times—or any institution with a whiff of liberalism, Republicans might just ignore them altogether. They aren’t shy about saying they support Trump or another Republican candidate. They simple do not want to engage at all.

There is no easy way for a pollster to adjust to this phenomenon. A large segment of the country could be, in the coming years, opting out of polling altogether. For pollsters attempting to gauge a general election environment or determine how Americans feel about a variety of issues, this is a deep and potentially existential threat. Compounding the problem is the number of Democrats enthusiastic about polls. Democrats, as a rule, appear to be far more willing to answer pollsters and share their opinions, skewing surveys even more. Democrats probably inflated the polls of a number of Senate campaigns in 2020 as they rushed to show their support for certain candidates.

Pollsters can only hope the eventual disappearance of Trump from the scene returns Republicans to the way they once were, fulminating about the liberal media while still answering a call from Quinnipiac. Anything is possible. In the meantime, I do believe journalists, pundits, and ordinary citizens should treat general election polls and survey opinions that attempt to track Republican viewpoints much more skeptically. Rosy projections for Democrats may not be so rosy.

This does not mean dismissing all polls of House and Senate contests or wailing “fake polls” at a television screen. What it does mean, rather, is weighing the reality of the Republican drop-off with any poll you come across in the media. Polls that show Biden easily defeating Trump or any 2024 Republican opponent should be taken lightly. We live in an extremely polarized country where election routs, on the national stage, no longer happen. There are few, if any, competitive states where Democrats can dominate Senate contests. If Republicans truly do leave polling behind, press coverage, at the bare minimum, will have to reflect this. We can’t pretend we live in the old world.