How Polling (Still) Gets It Wrong

Letitia James wasn’t supposed to be the Democratic Party candidate for New York State attorney general. Polling done just days before last week’s primaries gave Sean Maloney the edge. Another candidate, Zephyr Teachout, held third place in the polling. Instead, the results were as follows:

• James: 578,382 votes
• Teachout: 442,733 votes
• Maloney: 357,665 votes
• In short, polling not only failed to call the winner but also predict the final order of the three top candidates. The leading candidate placed third, and the supposed laggard put in a good showing.

All of which got us to wonder: how do polls get things so wrong? There are numerous examples of recent history, from President Trump’s 2016 surprise election to Brexit, on this count. But recent history such as Andrew Gillum’s Democratic Party win to vie for the Florida governor’s mansion (he was polling 4th) and the NY AG primary shows polling is still an imprecise science. At best…

In the case of the last cited poll ahead of the NY AG vote, the problem seems clear: too small a sample size at just 509 likely voters. Primaries tend to be low turnout events, so the polltakers likely felt they only needed a small sample to complete the mission. Big mistake, that.

But given how much is riding on the US midterm elections (control of the House… perhaps the Senate too) and what that may mean for capital markets, just shrugging away these failures isn’t really an option for investors. Instead, it pays to understand what can go wrong, as polling conditions expectations around election outcomes.

Here are three common problems:

• Elections with low turnouts (primaries and the upcoming midterms) are harder to sample in a poll. This isn’t just a numbers game like the NY AG race example above. The exact composition of the electorate changes every election cycle. So who do you poll, and what weight do you give to your raw data? [su_spacer] This was a clear problem in the Florida governor’s primary race, outlined here: https://www.vox.com/2018/8/30/17796308/florida-primary-andrew-gillum

• Despite all the theoretical advantages of “Big Data” analysis, political polling has not actually gotten any better over the last 40 years. The margin of error going back to the 1970s is 6 percentage points. It is the same today. Better technology is offsetting lower response rates, leaving the net result about the same. Not better. [su_spacer] For a geeky analysis of political polls, check out Nate Silver’s recent take: https://fivethirtyeight.com/features/the-polls-are-all-right/

• In the end, polling has to accurately assess three variables, not just a binary win-lose outcome: the turnout for each party (2, in the case of the US, many more elsewhere), how many true independents will show up, and how that sliver of the population will vote. In addition, some percentage of each population will not answer a poll question truthfully, and some respondents will not even end up voting.

Bottom line: society as a whole and investors specifically have come to expect that any numbers/data based analysis should reliably predict future reality. That’s a reflection of an increased faith in technology. And in many cases, it works just fine.

But when it comes to predicting elections all that technology has not helped produce more reliable assessments, so take any/every poll with a very large grain of salt.