Lessons from the Election?

November 11, 2016

No, I am not going to rail (or boast) about the election results. What I think we should look at carefully is the evident widespread failure of polling and of the predictions based on the political polls. By one estimate, 90% of all political polls were wrong. And interestingly, that is not new, not even in the past 12 months. Think back to Brexit, as well as to both several of the Republican and Democrat primaries, where polls and related predictions also failed badly.

So? It should serve as a caution to all of those involved in CI. Let me explain:

  • The polls included assumptions that past turnout and voting performance were a predictor of future performance. Didn’t the pollsters ever run into the disclaimer that the securities industry always uses: “Past performance is no guarantee of future results”?
  • Some of the analysts, when looking at the polls, ignored an academic’s recent study that, and I think I have this right, that the usual margin of error of 3 points for each side of the polls was, when getting closer to election, at 6 points, or twice as large as claimed by the pollsters. That meant that these late polls were, essentially, becoming more worthless as the election came closer. Did they react to that? No, they continued to predict based on what models had worked in the past for them. A real “blind spot”, wouldn’t you say?
  • There is some indication that the polling may have been impacted by the blanket refusal of some groups – or subgroups – to participate in the polling, or if they participated, to “blow smoke” at the pollsters. What did the pollsters do? Evidently they projected the behavior of these absent or underrepresented groups by reference to the behavior of rest of the electorate. There was no discernible effort to figure out why these groups were under-represented, how big they were, and what that meant to these data holes. By doing that, they failed to account for what former Secretary Donald Rumsfeld famously called the “unknown unknowns”. They assumed that what they had learned from those who could be massaged to cover those who refused to be polled. “Mirror imaging” of a dangerous sort?

For those of us in CI analysis, remember that what worked yesterday may not work tomorrow, what was true yesterday may not be true tomorrow, and that predicting what people, individually or in groups, will do is, to say the least, fraught with peril.


2 Comments on “Lessons from the Election?”

  1. Thanks for this excellent and timely post, John. Everyone in our industry should be concerned about this, and together, we can develop answers and solutions.

    I’ve cited you in my own initial comments, posted here: http://www.knowledgevaluechain.com/2016/11/12/bad-night-for-big-data/

  2. […] For those of us in CI analysis, remember that what worked yesterday may not work tomorrow, what was true yesterday may not be true tomorrow, and that predicting what people, individually or in groups, will do is, to say the least, fraught with peril.  […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s