Recalled-vote Weighting: The Most Important Methodological Challenge in the 2024 Polling

October 9, 2024

The polling is scary for a lot of people now. Mostly this is because so much of it is so close, and people fear that the polls are undercounting Trump supporters.

I am not sure the election is really as close as the polls suggest, and that is because I do not think the polls are undercounting Trump supporters. In fact, I think they might be overcounting for them. Most good quality pollsters appear to be using a technique known as “recalled-vote weighting” to create a frame for this November’s electorate. Pollsters ask voters who they voted for in 2020 and then weight the poll based on the actual 2020 vote margin in the jurisdiction (state or nation) they are surveying.

Source: New York Times

It may seem counter-intuitive, but recalled-vote weighting results in higher weights on the losing party from the last election. One reason for this is the well-known phenomena of voters misremembering who they voted for (often believing they supported the winning candidate when they did not) or whether they voted at all. In fact, Monmouth is using recalled-vote weighting this year and admits that 10% of their respondents who claim they did vote in 2020 did not appear to vote according to voter file data.

Pollsters who are using this technique are likely doing so because of the sensitivity they have over the polling misses in the last two presidential elections and the public reaction to that. Trump may be a uniquely problematic candidate when it comes to modeling the electorate and even eliciting responses, so it may be that there is no good way to adjust one’s methods to truly capture what is happening among voters and potential voters. Pollsters have to “pick their poison,” as Monmouth’s Patrick Murray has said.

It is possible that 2020 vote recall will be accurate this year, but if that’s true then it will be the first time this century. Nate Cohn, of the New York Times, notes that if we applied this technique to past polling the misses would have been significantly increased in the direction of the party that lost the previous election. This is why the Times/Siena College poll does not use the technique. If we look at the polls that do use it, the results look unsurprisingly like 2020 results. When we look at polls that do not, like Siena, we see things differently. One thing we see is a closer national race, but clearer leads for Harris in battleground states.

[Recalled-vote weighting] may make it harder to identify any changes since the last election. One reason is simply because recall-vote weighting mechanically forces polls toward the last election result. But even deeper than that, the technique is being used, at least in part, because pollsters don’t trust their survey results. It’s an attitude that can lead pollsters to treat surprising and unusual data as suspicious, rather than potentially newsworthy or insightful. This is understandable: I know I don’t always trust our survey results after the last decade of polling misfires. But when legitimate concern about polling morphs into weighting away otherwise outlying findings, it risks squelching any indication of anything that might be out of the ordinary.

Nate Cohn, New York Times

Florida: recalled-vote weighting may be muting rightward shift of the state electorate

Is it more likely the lurch rightward the Sunshine State took during the pandemic is sticking rather than reverting to a pre-2020 norm? A new poll from the New York Times/Siena College suggests that’s true. Trump is up 55-41 (+13 with rounding) in a state poll that is far different than others we have seen in this cycle. The 538 and Silver Bulletin averages are Trump +5. New York Times Poll Tracker has it Trump +7 (but that includes the new +13 poll).

Ironically, if this Florida result is correct - indeed, if Siena has been generally correct in its polling - then the Republican advantage in the Electoral College has largely disappeared. Harris will not need a three point of better national share of the popular vote to ensure an electoral vote victory. Nate Cohn notes that the shift they are seeing in Florida and New York (still an easy Harris victory, but potentially the margin will be half that of Biden’s) are enough to make the national popular vote a close one, but they do not shift the electoral vote at all. This is something I suspected was possible a few months ago, while Biden was still in the race.

Florida appears to be a solid Republican state now when we look at voter registration, which has strongly moved rightward in the past few years. That is less true of newer registrants who did not vote in the last election, but Republicans still outpace Democrats in that group. For this reason, the Times has concluded that a Trump +3 electorate - which a 2020 vote recall electorate would look like - probably does not exist in Florida anymore.

However, the Senate race remains close - and indeed, this Florida poll is an outlier at the moment - so it’s not really clear what is going on in the state. Perhaps Florida has the possibility of being a battleground state, or perhaps it is so solidly Republican now it is out of reach for Democrats. I am not sure we will have any clear idea until the votes are counted.

In general

The problem with the 2020 vote recall is that there may be significant changing dynamics in the electorate that favor Democrats this year. This technique is likely to miss a lot of new young voters, women voters, and voters of color who have registered to vote since the Dobbs decision and as a result of the Harris candidacy. We see this movement in the voter registration data across states. Florida is not an exception, but there are so many Republicans moving there and registering there that it mutes the phenomenon. Nevertheless, we hear pundits talk about Republicans having a voter registration advantage in many battleground states. That is only true if one only compares Republican and Democratic registration.

The early vote data we see so far in Michigan show women voting at 2022 rates and Black voters significantly outpacing their share of the vote in 2022 and 2020. The mail ballot data in Pennsylvania show Democrats with a 6.1% lead in returning ballots across all counties, not just statewide. This data suggests that in those states the electorate might look more like better for Democrats than it did in 2020. (Wisconsin does not report mail ballots by party so we can’t see any specific trends there, and the electorate is much more spread out there than in most states).

We are seeing large numbers of new voters register as independent or unenrolled in the battleground states. These are not likely to be Trump voters. Young voters are increasingly registering as unenrolled. The polling we have has such small sub-samples of young and new registrants that the error bars are way too high to have any crosstab results be reliable. However, the Harvard Youth Poll surveys young people exclusively. And in that poll we get a much better idea of who young voters are. They are more progressive on the issues than Democrats in general. And they are supporting Harris in large numbers.

So, what does all of this mean?

For the polling industry, it may avoid the public criticism of the past several years by largely using recall-vote weighting. It is possible that this election will indeed look like 2020, in which case the polling has a good chance of looking fairly accurate. It is possible Harris will win by slightly (probably not significantly) larger margins that the polling suggests, and frankly few people will care that the polls had the race closer (if the non-reaction to the 2012 polling miss is any indicator). If the electorate is more pro-Trump than 2020 and he wins – especially if he wins by several points – that is where the pollsters may suffer the backlash. While not out of the question, that appears to be the least likely scenario right now. So, there is little incentive for pollsters using this technique to abandon it now.

For the rest of us, what it means is the polls are likely to be wrong this year because the electorate is probably going to look very different than the one pollsters are modeling. The overall effect on the popular vote may not be noticeable; Harris might win a close vote. But chances are that the electoral vote will favor her if she wins the popular vote. It does not appear that she will need a several-point margin in the popular vote to win enough electoral votes. It’s less clear to me how that might impact Trump if he wins the popular vote because I still do not think it is possible for him to do so. I think his only path to victory is an electoral one; that if he accomplishes that he will still have lost the popular vote.

This should give people hope that Democrats do not have to run up the score to win the election. But they will still have to run up the score to undermine the election interference attacks that Republicans plan to make. A Harris victory of five points nationally and 330 electoral votes will be much harder to undermine than a narrow popular vote win with barely 270 electoral votes. Running up the score has to remain a focus for everyone who supports democracy. We cannot trust the guardrails to hold in a close election this year. They might, but we are stressing the system too hard these days.