How did the polls fare in Election 2015?

On the whole the polls did well in Election 2015.

Below is a table displaying the difference between the final polls from different firms and the actual preliminary vote count. I subtracted the poll number from actual vote shares. Negative values mean the poll underestimated party support while black represents an overestimate. Angus Reid provided polling numbers representing both eligible voters and those they deemed to be likely voters.

Very few numbers are outside the margin of error specified by the pollsters (note that margin of error does not really apply to online pollsters).

What is striking is that the two most accurate surveys were both conducted on the day before the October 19 election, which strongly suggests that late movement to the Liberals was a significant factor in the outcome. Also note that the likely voter model of Angus Reid performed less well than their eligible voter model. This may partly reflect the higher turnout we saw on election day.

The numbers above deal only with the national scene. However, seat outcomes depend much more on regional numbers, which being based on smaller regional samples are prone to greater error. However, I think the table below suggests that even in the regions the polls did perform well.

Note that Nanos has an advantage here. It would have a slightly smaller total error because it reports on one fewer region (the Prairies rather than treating Manitoba-Saskatchewan as one region and Alberta as another). Nevertheless it is clear Nanos performed well.  I have also omitted the Bloc from this table. The numbers here of necessity are all absolute values (the difference between the regional vote shares and the poll numbers) as it was the only practical way to aggregate them. This is why all are positive values and there is no red as in the table above.

Mainstreet Technologies finished behind Forum and Nanos I think mainly because the poll was finished earlier. However, it had the largest sample size of any pollster at 5,546, partly accounting for a strong overall performance.

My conclusion is that the polls did well this time despite the obvious challenges confronting them. On those challenges, this article by Donna Dasko in the Globe and Mail is a must read for anyone wanting to understand the dilemmas confronting pollsters today.

They did get it right this time but the possibility of a future fiasco as we was in British Columbia in 2013 and Alberta in 2012 remains.

What did not do well were the seat projection models like my own. This is a subject for a future post.

A Liberal minority is in prospect

We highly personalize politics now and many will attribute a Liberal government, most likely a minority, to Justin Trudeau.  However, his leadership numbers, while they have gone up substantially over the course of the campaign, have trailed party preference. This is neither surprising nor new. It has happened before, notably in 1993 when Jean Chrétien went from yesterday's man to PM over a two month campaign. This result as in 1993 appears signficantly driven by negative considerations on the part of voters. Regardless, Trudeau will become prime minister

Personalities were front and centre yesterday when Stephen Harper attended an event in Toronto organized and sponsored by the Ford brothers, Rob and Doug. Normally the ever astute Harper would avoid relying on the polarizing Ford brothers, but times for him are desperate. He hoped for some boost from the Fords that might pull out a few extra blue collar, low education, low information Tory voters from Ford Nation.  By doing the event he effectively conceded that his Finance Minister Joe Oliver would lose his seat. A significant percentage of Oliver's riding was one of the areas of Toronto that most strongly supported John Tory and its residents have nothing but distaste for the Fords so you know Harper has given up on it. It was symbolic of the campaign closing that Harper's campaigning with the Fords was about despair rather than hope (but it did give the cartoonists a field day).

My estimate of likely seats won based on average of the closing polls is below:

I think all seat projections need some qualifications that can't be quantified. Others such as 308 present seat ranges. I regard such error estimates as meaningless. Probability error for polling results is based on statistical theory (assuming the sample is genuinely random). Error ranges for seat projections make no real sense to me (with one exception I won't go into).

Seat projection is as much art as science and errors are likely to be greater in some circumstances, some of which are present in this election:

• First, the Liberals are going from third to first and large changes may break in significant ways from previous voting patterns. 
• Second this election is fundamentally about strategic (better described as tactical) voting - anybody but Harper. Strategic voting may elect a slightly larger number of NDP MPs (all would be incumbents) across the country than the trend would suggest. In my 1999 study on this topic the overall trend suggested just one New Democrat should win. Nine were actually elected. 
• Third, if micro-targeted strategic voting is truly effective, it may mean fewer Conservatives will win than the trend suggests.
• Fourth, Quebec broke radically from its past last time and now seems to be experiencing a series of idiosyncratic shifts that may well produce a number of surprising outcomes.
• Fifth, the Liberal surge is strong enough that it could produce a few perverse effects, letting Conservatives win in a circumstance where a New Democrat might otherwise have been successful and the more appropriate strategic choice.

I have previously written that the campaign hinged on the mid-September niqab announcement.  As a counter-factual what if  we suppose there had been no such development. Instead on the same day the Gagnier story had broken. What might have been the impact? Now I actually I think that from what we know of the Gagnier affair, it was not likely going to have much impact, but lets presume it had been more scandalous than now appears. Might it have overwhelmed the other factors that had helped the Liberals up to that point. We will never know, but it is clear that sheer chance can have an outsized impact on election outcomes. What is not due to chance is the sheer size and scope of the distaste that most Canadians have for the Harper government. That is the most important factor determining the outcome of this election.

The story of Campaign 2015: The Niqab and Anyone but Harper

A version of this post can be found at iPolitics

The 2015 campaign and its likely outcome can be boiled down to two factors: the Federal Court of Appeal decision on September 15 that allowed Zunera Ishaq to be formally sworn in as a Canadian citizen while wearing a niqab, and the fact that while a significant percentage of the Canadian population was content to see either the NDP or Liberals win, they wanted more than anything to see Stephen Harper leave. They were ABH - Anyone but Harper.

Widespread feelings of antipathy towards Harper have fundamentally framed the choice — but until mid-September it wasn’t clear which of the other two parties was going to be seen as the appropriate alternative. The polls then were telling us that it was a three-way national tie.

That’s where the niqab came in. A part of Quebec’s francophone population, which up to that point was willing to vote NDP, had a strong, visceral, negative reaction to the niqab. It was powerful enough to push large numbers to abandon the NDP for the Bloc and the Conservatives (Take a look at the Bloc's ad on this). Tom Mulcair defended the right of women to choose what they want to wear, and paid a price.

There was an early impact. Any new development takes a few days to register with the electorate (keep this in mind when thinking about the Gagnier affair, and don’t expect to see any significant impact before Sunday in the polls). Shortly after September 15, the NDP rise in Quebec stopped and then started to fall — fast. After a short delay the impact grew and was reinforced by the first French TV debate, but it was clearly visible before. The NDP decline began to be reflected in national polls — again, prior to the debate. The Liberals had the same position on the niqab but managed to escape the backlash largely because their Quebec support is disproportionately anglophone and allophone.

You can see it in the following two graphics — the first is the Quebec polls from September 1 to mid-October. (Note that I use a three-poll moving average to smooth the natural fluctuations you get from polling).

And with a short delay, a gap between the Liberals and the NDP opens up in the national polls.

The gap gave the clear signal to anti-Harper voters. The Liberals and Justin Trudeau have been the clear winners because Quebec turned against the NDP as a consequence over the niqab — a bitter irony for Stephen Harper, who tried to exploit anti-niqab sentiment and had been running anti-Trudeau TV commercials in constant rotation.

There has long been evidence of strong dislike of the Harper government among many Canadians. It’s why you hear so much talk of strategic voting. The expectation of many advocates is that such voting would efficiently topple just enough Tories to do Harper in. Whatever its micro impact in individual constituencies, it’s clear that we’re seeing the effect of strategic anti-Harper positioning on a broad national scale.

But what about the rest of the campaign — is this really all that mattered? There were some Liberal gains on the NDP between mid-August and mid-September in Ontario that could be attributable to their efforts to outflank the NDP on the left, but it might also have been simply a movement back to the pre-Alberta election polling norm. Such movements did occur in Western Canada a little earlier.

The debates (except in Quebec) were generally a fiasco — bad television with tiny audiences largely composed of the politically-committed. Trudeau did exceed expectations in the debates, which helped him in news coverage, while Mulcair fell short. Trudeau had an effective ad (the one on the escalator) while the NDP made the mistake of not advertising early enough — campaigning as a frontrunner when they really weren’t that far ahead.

But the real explanation for the pattern this election has taken is really quite simple: the niqab controversy wound up signalling to Canadians how they could best express their opposition to Harper.

And the 2015 election has been about Harper, his protests to the contrary notwithstanding. Its zeitgeist has been captured well in Michael Harris’s acerbic columns in iPolitics. Canadians can expect to wake up to a very different House of Commons come Tuesday morning.

Who got it right in 2011: Pollsters and Accuracy

This has been cross posted at iPolitics

It can be difficult at the best of times to make sense of the flood of polling data issuing forth from Internet and airwaves. And there’s a fair amount of social media buzz out there about which pollster is getting it right in this election.

Two of Canada’s major polling firms, EKOS and Nanos, have been reporting different parties in the lead — the Liberals on the part of Nanos over the past week, the Conservatives in the case of EKOS (although the latter’s most recent release gives the Liberals a slight lead that puts them in a statistical tie with the Conservatives, given the margin of error).

So who is getting it right? We won't know for sure until we take the ballots out of the box on Oct. 19; until then, we can do little more than speculate.

What we can do is look back at earlier elections. In 2011, when it came to national numbers, the firm with the fewest errors was Angus Reid, a repeat of its 2008 performance. Reid finished just ahead of Nanos, which in an earlier incarnation as the firm SES had been closest to the mark in the 2006 election.

The table below compares the performance of polling companies in 2011 with respect to national vote shares:

No one pollster gets it right all the time. Compas, no longer providing polling reports, was last on this list in 2011 but the most accurate in 2004 (with Ipsos and Léger just a whisker behind). EKOS ranked relatively low in 2011 (to its credit, EKOS performed a post-election evaluation of its performance). However, EKOS was closest to the election result in 1997, while Environics had that honour in 2000. The title has been widely shared.

One interesting pattern easily discerned from this list is that most firms underestimated how well the Conservatives would do in 2011. On the other hand, Conservative support was widely overestimated in 2004 and Liberal support was underestimated. As a result, we had election night surprises: a Conservative majority in 2011 and a stronger than expected Liberal minority in 2004.

While getting close to the actual result is important to firms, in terms of election outcomes the national number is something of a beauty contest. When it comes to seats, what’s happening in the provinces and regions is what really counts, and it’s difficult and expensive to obtain accurate polling numbers that capture all of Canada’s political diversity.

The table below takes the regional numbers from the pollsters above (except Compas) and then creates an average of errors in the regions for each party. A total lets us compare them. You will see the order of finish is somewhat different.

While the rankings differ from the national picture, that’s not what is most important. Many values are relatively large. In a first-past-the-post system, a small deviation of two or three points one way or the other can have a significant impact on ridings won or lost.

The under-estimate of the Conservative vote led directly to forecasters’ missing the impending Harper majority. Going into voting day, Harper had an average 6.2 percentage point lead over the second place NDP — but wound up with a nine point lead. Of particular significance was a 9.4 percentage point poll lead in Ontario that wound up on voting day as a 19-point lead.

Since 2011 we have had polling fiascos in Alberta in 2012 and British Columbia in 2013. The polls in these two elections created strong expectations of a win by the opposition that never materialized. This year, despite the radical change the Alberta election produced, the polls were generally accurate in forecasting the outcome (Léger Marketing was closest to the mark).

Methodologies have changed and diversified. For example, we have traditional telephone polling from Nanos and Environics, computerized telephone polling from EKOS, Forum Research and others, online surveys from large Internet panels from Ipsos, Abacus, Léger Marketing and Angus Reid and others. Resistance to answering polls has also increased, adding to the variation in results and the uncertainty.

A new Internet methodology that has seen some use in the United States is the Google Consumer Survey, which is a short survey that randomly pops up on computer screens. According to polling guru Nate Silver, a Google survey was the second-most accurate poll in the 2012 presidential election; this could be a glimpse of our polling future.

I am old enough to remember provincial election campaigns with no polls. However, polls have proliferated as never before, as have the individual riding surveys that attempt to drill down into the dynamics of close-fought local races.

Will strategic voting make a difference to the outcome of the 2015 election?

This has been cross-posted at iPolitics.

Since the 1960s, the Conservative party — in its various identities — has been moving steadily to the right. This has opened a widening gap between what is now a very right-wing party and the various parties of the centre and left.

Most voters do not have strong partisan alignments, but a large number hold values and political views that place them a considerable ideological distance from Stephen Harper. They have a strong incentive to vote strategically and they likely will. Their impact on the election could be significant.

The first person in Canadian politics to embody Harper’s ideological outlook was former Ontario Premier Mike Harris (several of Harris’ former ministers became part of the Harper cabinet, including Tony Clement, the late Jim Flaherty and John Baird). In 1999 Harris was the first major target of strategic voting.

In the 1999 Ontario election, Harris was re-elected. Nonetheless, that election saw a wave of strategic voting that had an impact. Harris received a vote share almost identical to that received by his party in 1995. However, the same voting support that won the PCs 63 per cent of seats in the legislature in 1995 secured just 57 per cent of Ontario constituencies in 1999. A detailed analysis of voting patterns I prepared after the election identified at least eight of a total of 103 seats where strategic voting accounted for the defeat of the PC candidates (seven of those ridings were won by Liberals, one by a New Democrat). In five more constituencies this voting pattern narrowly missed defeating Tories. The Dalton McGuinty-led Liberals were the principal beneficiaries as the NDP continued to be hobbled by the unpopularity of the Bob Rae regime defeated in 1995.

The 1999 precedent suggests that similar behaviour in the 2015 federal election could be a factor in up to 40 constituencies. Let me draw attention to some ridings on the front lines of the strategic voting war.

In Winnipeg, the riding of Charleswood-Assiniboia-St. James-Headingley has long been assumed to be safely in the hands of former Harper cabinet minister Steven Fletcher. However, Liberal support is up dramatically in the province and a recent province-wide survey by Manitoba pollster Probe Research reports the Liberals and Conservatives tied at 39 per cent each. In 2011 the Conservatives won 53.5 per cent of the Manitoba vote versus 16.6 per cent for the Liberals.

Much attention has been paid to the possibility of three south-end Winnipeg ridings switching from Conservative to Liberal, but little to this west-end constituency. However, my seat projection model suggests this seat is ripe for the picking by the Liberals and strategic votes from the approximately fifteen to twenty per cent of the vote won by the NDP and Greens last time could make the difference. The fact that the NDP recently replaced its candidate here makes the chances of it happening even greater.

In Saskatchewan, where the NDP last won a seat in the 2000 election, there has been a similar decline in Conservative support from the 56.3 per cent they won in 2011 to somewhere in the low forties, while both NDP and Liberal support is up. Since the last election a significant redrawing of boundaries has opened up the possibility of deep erosion of the near-monopoly of seats held by the Conservatives (the exception in 2011 was Liberal Ralph Goodale).

Several New Democrats could end up taking seats in Regina and Saskatoon, possibly shutting out the Conservatives. A recent riding poll in Regina-Lewvan placed the NDP six points back of the Conservatives (note of caution: constituency polls don’t have a great track record and have a difficult time properly locating the residences of cellphone users). If enough Liberal supporters vote strategically in Saskatchewan’s cities for the NDP, several Conservative-held seats could fall.

In 1999, several strategic voting organizations set up in Ontario to endorse candidates. However, their organizational strength and financing paled in comparison to the political parties; their efforts were too little, too late. Political parties are highly organized in every constituency they think they can win. They knock on every door and make direct appeals to voters by phone. They build up databanks of “positive” supporters and pull them out to vote. Strategic voting advocacy groups simply cannot hope to match this.

Strategic voting in 1999 was a grassroots phenomenon on the part of highly-motivated voters. There are strategic voting groups active in this campaign with the resources to do their own polling — at least one is planning to call households on election day — but my guess is most voters who want to cast a strategic ballot will simply do it on their own, as they did sixteen years ago.

There are two weeks left until voting day, and the split in the polls that has opened up between the Liberals and the NDP suggests the Liberals will be the principal beneficiaries of strategic votes. But voters need to do their homework. Canada is a large and complex country. Those who feel strongly about getting rid of Harper ought to take their local, regional and provincial circumstances into account before making a decision.

No party should automatically get a strategic voter’s ballot if the point of it all is to replace the Harper government as efficiently and effectively as possible. The Harper government could be dispatched on October 19, and strategic voting could well play a role.