“Shortcuts lead to blind spots.”
Fernand Amandi, IOP Margin of Error Conference, February 19, 2021
Last week, the Cook Political Report and the University of Chicago’s Institute of Politics held a conference on what we’ve learned from polling during the 2020 cycle. You can watch the sessions at the IOP’s YouTube channel here. There was a lot of interesting information on polling presented at this conference. Today I am going to cover just a few of the topics that are of particular interest: reading campaign polls; modeling problems, and; methodological issues, particularly how they relate to specific demographic categories.
In a future post I am going to discuss the differences between campaign polling, media polling, and analytic polling. These differences are important and merited some discussion at the conference, but are worth a separate post.
Numbers to watch in presidential polls
We learned two simple – but important – things about presidential polling from the chief pollsters for Biden (John Anzalone of ALG Research) and Trump (Tony Fabrizio of Fabrizio, Lee, & Assoc.). First, the leading indicator for an incumbent president running for reelection is not his or her poll numbers, but job approval numbers. According to Fabrizio, it is very difficult for an incumbent to run ahead of job approval numbers. Historical numbers support this. This will come as no surprise to readers of this newsletter. We discussed that during election. However, it is notable that the incumbent president’s pollster in 2020 was thinking the same way.
Second, the leading indicator for a Democratic presidential candidate is that candidate’s topline poll number. According to Anzalone, “Democrats get what they poll.” The margin is not helpful. So, while many people thought presidential polls were wrong because the actual margin of the election was much closer than much of the polling suggested it would be, they were in fact accurate. At least for the campaign’s purposes. Biden got what he was polling. The final FiveThirtyEight polling average gave Biden 51.8% of the vote. The actual result was 51.3%. State results were similar.
According to Anzalone, “Margins are the headlines.” What he meant by that is that margins are not helpful for campaigns, but they make good headlines in newspapers. The campaign wanted to know what Biden’s number was; the margin did not matter. As it turned out, Biden’s number was spot-on. It was Trump’s number that was wrong, and thus the margin was wrong. What this showed - and Fabrizio concurred - was that late-deciders largely went for Trump, but since Biden had a majority of the vote already those voters could not change the result (in the presidential race; it could have made a difference in down-ballot races).
Modeling problems
Anzalone claims pollsters have a modeling problem, not a polling problem. However, this problem impacts polling because polling uses models. If you overlay a bad model with a good survey design, you will still have problems.
Ann Selzer, the Iowa pollster who famously called the 2008 Democratic caucuses for Obama and also got the 2020 Iowa results right, is dismissive of likely voter modeling. She uses a method she calls “polling forward,” in which she lets the data lead her to conclusions. Before collecting data, she asks voters whether they plan to vote and accepts their responses. Selzer does not think likely voter models — which she refers to as “polling backward” — are helpful. What most pollsters do (and that was true of other panelists in this conference) is look back to past elections to project future electorates. Selzer says this does work — until something changes. Likely voter models are not good at seeing new voters. This is why other pollsters missed Obama’s Iowa victory in 2008.
Selzer suspects there is “a lot of guessing” going on in building likely voter models. Pollsters do not typically make public the specific nuts-and-bolts of the models they use. Consequently, it is difficult to understand exactly what goes into any pollster’s idea of a likely voter. Some use strict voting history, some do ask but also cross-check with voting history, and some use a “propensity score” in which they assign points to things like voter history, if voters know where their voting station is or what precinct they live in, etc. For Selzer, the easiest method is also the best. Just ask your respondents and trust the answers.
Methodological challenges
There seems to be a consensus that mixed-mode methods are best in trying to reach voters. However, there was disagreement over what the mix of modes was best in reaching particular groups of voters. There was agreement that it was difficult to survey voters with low trust in institutions. Anzalone claims that they missed these low-trust voters, but since they have such low trust in institutions “we’re probably never getting them on the phone.”
A lot of these low-trust voters are also first timers (or also voted in 2016), according to Fabrizio. They are not Republicans, but Trump supporters. In fact, they might never vote again. After all, they didn’t vote before 2016 and they didn’t vote in 2018 when Trump was not on the ballot. However, these voters were never persuadable; they will never vote for Democrats, according to Anzalone.
Campaign pollsters — and perhaps some media pollsters — are already moving to incorporate the text-to-web format, which allows a voter to receive a text and then follow a link to a webpage to answer questions at their leisure. Fabrizio claims that with this mode they are reaching more voters who are conservatives and Republicans. It has also helped reaching the “hardcores.”
Fabrizio calls IVR (robocall polling) the “bane of my existence.” The method works by automatically dialing until it fills up its demographic categories. Pollsters see a difference in results depending on whether respondents are using landlines or mobile phones, but IVR cannot reach the latter. Legally, autodialers cannot be used to call mobile phones.* Phone interviews must be done by humans, which is time-consuming and expensive. The reason for this law dates back to the dawn of widespread cell phone use when phone companies charged for every call made and received, so getting unwanted calls would cost the recipient money. That’s no longer the case, but the law has not caught up with reality. This means that IVR misses large portions of the electorate who do not use landlines.
Both chief campaign pollsters agreed that claims that working class whites were missed is not correct. The problem is more nuanced that that. There appears to be more of a disconnect between hourly and salary workers among low-educated white voters. There are many low-education workers in professional occupations these days. The other disconnect was between urban and rural. Low-education voters were more likely to respond in urban areas than rural areas. The challenge here seems to be getting responses from rural voters who do manual labor (and are paid hourly).
There were really interesting observations and important recommendations from panel on Race, Demographics, and the Future of the Electorate. Fernand Amandi (Bendixen & Amandi International) claimed that the “gold standard” for Latinos is the phone, but that met with some disagreement. Matt Barreto (LD Insights) insisted that is only true for older Latinos, not the youth. Both agreed that online polling is problematic with Latinos. This is because the digital divide problem remains real – especially with seniors and foreign-born. There will be a time when that will no longer be true, but in the foreseeable future it will have an impact on reaching Latino voters.
To understand the voting trends among Latinos and AAPIs (Asian-Americans and Pacific Islanders), it is necessary to segment the vote. This is especially true for AAPIs, which include folks from backgrounds as diverse as India, Cambodia, Vietnam, China, Japan, and The Philippines. However, we do see different voting trends among Mexicans, Cubans, Venezuelans, and Colombians. It is more difficult to segment among Latinos by making assumptions about names than among different AAPI populations, but even with the latter names are often not helpful. What is needed is more bilingual and multilingual interviewers. Folks who not only can speak the language but can think and speak in real time and have the cultural competence to understand the nuance of conversation in non-English languages. This is expensive to do. Another problem is the lack of BIPOC folks at the design stage of polling. They are not often in the room when important methodological decisions are made.
According to Barreto, the AAPI population is growing faster than Latinos (which itself is growing fast), but a smaller part of the electorate overall. Almost 70% of AAPI are foreign-born compared to 30% for Latinos. AAPI voters can swing by as much as 20 points from one election to the next. For other racial/ethnic categories it is five points or less. Generational issues and differences are very important to AAPI. Smaller groups are always harder to poll but they are important political actors nonetheless. You can try oversampling, but it is hard to know who fits in any particular segmentation – names don’t always help – particularly for Latinos.
Finally, Amandi noted that men were “a surprise.” Across Black, Latino, AAPI populations, men all supported Trump in higher numbers than expected. It was still small, but large enough to take Texas and Florida out of contention for Biden.**
The general consensus seems to be that it is possible to do accurate polling even in an era of reduced response rates. The challenge is financial. To engage in mixed modal contacts with a strong complement of bilingual and multilingual interviews gets very expensive quickly. This may be the most difficult challenge facing pollsters today.
However, Nate Cohn (The Upshot) left us with this observation. None of the problems discussed at this conference are new. In fact, the same conference in 2016 could have covered all the same points. Pollsters have not rigorously researched any of these problems from hypothesis through conclusion, and thus we still don’t have good answers to issues such as the social trust phenomena. That, Cohn says, is a challenge that left unmet will result in us raising the same concerns in a conference four years from now.
—————
* Autodialers can be used for mobile phones, but a real human has to interview the respondent as opposed to an automated survey.
** This is my claim, not any of the panelist. I am not suggesting that Biden would have won Texas or Florida, but considering how close it would have had to have been in either state for a Democratic victory the increased numbers of non-white men voting for Trump likely made it impossible.