More on the B.C. polls

The invaluable Pundit's Guide points us to an interesting and revealing polling experiment that compared telephone IVR (interactive voice response) polls to online polls.  The conclusion according to a release from Campaign Research, which conducted the experiment, is that IVR beats online:

“I question the reliability of voter intentions derived through web based on-line panels” said Campaign Research Principal, Greg Dunlop. Further he said “We believe that IVR polls conducted properly are much more representative of the voting population.”

One notable finding was that an IVR poll conducted on May 13 just prior to election day had the Liberals ahead by 41.6 to 38.0 whereas an IVR poll conducted May 10-12 had the NDP ahead 41 to 38.6.  The online poll reported an NDP lead of over 5 points. This suggests that there was some last minute movement to the Liberals.  Most polls conducted during the campaign, however, were online polls.

Another commentary from Ekos President Frank Graves`s, whose poll projected the NDP vote accurately but underestimated the Liberal vote, has written a comprehensive self-critical assessment, noted among other things that turnout is a critical factor in explaining the difference between polls and results:

The pollster sweepstakes that surrounded the nearness of final poll to election outcome once made a lot of sense. In the past, most people voted and the relatively small minority of those who did not vote were not systematically different. So the yardstick made sense. Neither of those conditions applies today.

In British Columbia, the electorate can be divided into two roughly equally sized groups – those who voted and those who did not. This is fairly typical of recent provincial elections. If the non-voters are very different than the voters, then a poll can be an accurate measure of the population of all eligible voters but a flawed predictor of the election outcome.

We have now seen significant differences between polls and election results in B.C. in 2013, Alberta in 2012 and in final polls before the 2011 federal election.  In particular, on average the polls underestimated the Conservative vote by about six percentage points in Ontario in the 2011 federal election, which by itself accounted for the difference between minority government and a Conservative majority.

The Campaign Research study, which provides a reasonably full accounting of its methodology nonetheless does not say how large the online panel was from which its sample was drawn either nationally or in B.C. However, it does say: "Web-based polls are not probability samples as the population that the survey sample is drawn from is, in itself, a non-random sub-sample derived through panel recruiting."

In the U.S. Gallup, the traditional gold standard in polling, is undergoing a critical review of its failures in the 2012 U.S. election.  There is clearly a broad trend here.