When polls don't agree - Ontario and Manitoba

One can usually gauge political trends relatively easily.  Polls vary, but they don't tend to contradict one another dramatically, so one sits up and takes notice when this happens.  There have been a couple of interesting recent examples involving Ontario politics and Manitoba politics, two provinces that will be holding elections a year from now within a week of each other.

An Ipsos-Reid poll released on August 21 received considerable attention because it had the Ontario PCs ahead - 36% to 35% for the McGuinty Liberals and 18% for the NDP. However, a one percent difference is really a tie given the inherent uncertainties of polling, the margin of error, etc.  A poll released by Angus Reid on September 28, just five weeks later (too soon for any actual opinion shift to take place), suggested a very different outcome: an eleven point lead for the PCs, 41% to the Liberals 29% and 22% for the NDP.  The difference matters even more in terms of seats.  TC's forecast model says the Ipsos poll would produce a Liberal minority government of 51 seats, the Angus Reid poll a large PC majority.  They are so different that one can say with confidence that one of these polls is wrong.  With no immediate election, however, we will never know which one it is.

Perhaps we might get some clue as to what to believe from Manitoba.  There, three polls have been released in recent weeks and a similar pattern emerges.  The opposition PCs were reported on September 21 by Angus Reid to have a 15 point province wide lead over the governing NDP 49% to 34% with the Liberals in third at 12%.  It was preceded by a Viewpoints Research poll conducted Sept. 7-15th that had the NDP one point up on the PCs at 39% to 38% with the Liberals at 14%. This survey was followed by a Probe Research poll on October 7th that had the PCs at 42% and the NDP at 40% with the Liberals at 12%.  Again we would have different election outcomes.

Despite the PC lead in the Probe Research poll, the concentration of PC support outside the City of Winnipeg and its weak performance inside would produce an NDP government in a new election - the PCs lead the NDP 53-32 in rural areas but trail the New Democrats 46-35 in the city. TC estimates that the Probe poll would produce a legislature with 32 New Democrats, 23 PCs and 2 Liberals; the Angus Reid poll would produce a PC government with just 30 seats (despite their large overall lead) to the New Democrats 25 and the Liberals 2.

A key difference between the surveys is that Angus Reid does online polling while the others use traditional telephone methods.  Online surveys are relatively new methodology. TC thinks they need to be seen as experimental.  Reid uses large panels (they describe some of their methods here) recruited at least in part through internet ads (such as this).  I have heard that their panel is about 100,000 in Canada.  However, it is not clear that this gives them the truly representative sample they need to properly capture public opinion. And one wonders how large their Manitoba panel can be. 

Angus Reid (or Vision Critical as it is named on its web site) has been close on some election outcomes, including in 2008 in Canada, but they have also had some bad results. Earlier this year they managed to get the order of finish wrong in the UK general election. Reid's final UK poll results are here and the final election results here.  But what explains the apparent Tory tilt in the Manitoba and Ontario polls and the disagreement with traditional pollsters? While the Manitoba and Ontario numbers are suggestive, the limited poll set here can't really tell us anything conclusive.

We know well the methods of the telephone polls but what of the online polls? Their methods remain relatively opaque, and they still haven't established a reputation for accuracy and reliability.  While Vision Critical is not included in his assessment, Nate Silver rated online pollster Zogby the least accurate polling company in a review of the accuracy of dozens of US firms. Éric Grenier of the blog threehundredeight.com estimates the "house effects" (or partisan tilt to put it another way) of polls.  His estimate (top poll in table) suggests Angus Reid's polls are favourable to both the Conservatives and the NDP.

TC puts more stock in polls conducted using the telephone and remains skeptical of online polling. I would therefore see the overall races in Ontario and Manitoba as close and still competitive.  Nevertheless, the NDP in Manitoba will be looking for a fourth mandate (with a new leader), and the Ontario government a third. Both situations imply a change in government even if it is not yet clearly evident in the polling.