Lessons from the Election?Posted: November 11, 2016
November 11, 2016
No, I am not going to rail (or boast) about the election results. What I think we should look at carefully is the evident widespread failure of polling and of the predictions based on the political polls. By one estimate, 90% of all political polls were wrong. And interestingly, that is not new, not even in the past 12 months. Think back to Brexit, as well as to both several of the Republican and Democrat primaries, where polls and related predictions also failed badly.
So? It should serve as a caution to all of those involved in CI. Let me explain:
- The polls included assumptions that past turnout and voting performance were a predictor of future performance. Didn’t the pollsters ever run into the disclaimer that the securities industry always uses: “Past performance is no guarantee of future results”?
- Some of the analysts, when looking at the polls, ignored an academic’s recent study that, and I think I have this right, that the usual margin of error of 3 points for each side of the polls was, when getting closer to election, at 6 points, or twice as large as claimed by the pollsters. That meant that these late polls were, essentially, becoming more worthless as the election came closer. Did they react to that? No, they continued to predict based on what models had worked in the past for them. A real “blind spot”, wouldn’t you say?
- There is some indication that the polling may have been impacted by the blanket refusal of some groups – or subgroups – to participate in the polling, or if they participated, to “blow smoke” at the pollsters. What did the pollsters do? Evidently they projected the behavior of these absent or underrepresented groups by reference to the behavior of rest of the electorate. There was no discernible effort to figure out why these groups were under-represented, how big they were, and what that meant to these data holes. By doing that, they failed to account for what former Secretary Donald Rumsfeld famously called the “unknown unknowns”. They assumed that what they had learned from those who could be massaged to cover those who refused to be polled. “Mirror imaging” of a dangerous sort?
For those of us in CI analysis, remember that what worked yesterday may not work tomorrow, what was true yesterday may not be true tomorrow, and that predicting what people, individually or in groups, will do is, to say the least, fraught with peril.