Posts
Much though I, like others, was fascinated by the polling numbers, overall they were not nearly as accurate as we should expect from polls, and the publication of polls themselves draws too much journalistic attention and energy better spent on issues raised during those contests such as inequality, which did get debated to some extent in Toronto Centre. As others have noted, this does a disservice to voters and might have affected the outcome in at least one constituency.
So just exactly how well did the polls do in these contests. It is completely measurable. We can compare the claimed margin of error for the surveys with the actual differences between Forum's final polls and the actual election results.
First a methodological note: the expected error for values that fall towards the extreme end of a range (in this case 1% to 100%) is smaller than for values closer to the middle. That means as a practical matter the parties that won only a small share of the vote almost always fell within a poll's claimed margin of error. I have therefore excluded the Green and Other Categories from the tables below as they don't really tell us anything about how well a poll performed. Among the major parties, there were occasions where a party received less than 10% of the popular vote in a given riding. I found only one case of party getting less than 10% (the NDP in the provincial Ottawa South by-election) where the error between the actual outcome and the poll number exceeded the claimed margin of error. It is the major contestants that are of principal interest, those likely to finish first or second in an electoral contest. That is where error truly matters. Among those the Forum polls did not do well.
The bolded numbers in the Difference table are outside the margin of error. What we see is that of the fifteen outcomes in the table above, the number exceeds the margin of error in seven instances. In one additional case, the NDP in Etobicoke-Lakeshore, their 6.1% vote share (7% in the poll), while within the margin of error, was too small to be important. This is a poor performance. In addition, Forum's numbers accurately anticipated in the winner in only three of five by-elections, one of which, Windsor-Tecumseh, was a runaway victory for the NDP.
Again, in examining the polling in the federal by-elections we find four outright errors among thirteen outcomes. In addition, the Conservatives in Bourassa and Toronto Centre, and the NDP in Brandon-Souris and Provencher had small vote shares (under 10%). So the fact that all these cases in the Difference table fall within the margin of error is not significant.
The polls were better than throwing darts; the poll numbers were reasonably close in Bourassa and Toronto Centre, but not nearly good enough. Overall Forum's polls were certainly weak enough that no Forum poll released during this most recent set of by-election campaigns deserved a big headline, and none in the future should be treated by the media in any fashion except cautiously and skeptically.
