How the riding by riding forecast performed.

I have now compared the predicted winner in my seat forecasting model to the actual winner in each constituency (on the basis of the preliminary results).

I have used the actual provincial vote shares as the input into the model. There is considerable confusion about seat prediction vs the polls. I find that it is common for many to confuse polling error with the error in seat prediction. I used the actual election day vote share because if the polls were perfectly accurate this is what one would get. It is almost impossible to predict the outcome in every riding accurately even with the perfect poll, so any forecast model will have error, but one should get the best results by using the most accurate polling data available, which is, by definition, the actual vote share.

Many still wonder what use this is since it comes afterwards, but almost all who read the polls during an election make an inference about outcome so I think seat projection models have a role to play. As we compare the poll numbers with the actual vote shares in an election, so we ought to compare the actual outcome in each riding with what the model predicts. When the final validated results are available in a few months I will compare the difference between my predicted and the actual percentage outcome for each candidate in each constituency.

In making predictions I do not calculate a predicted outcome for the three territorial ridings. In reporting results I simply assume that the incumbents there will be re-elected and report out a total of 308.

For the 305 constituencies using the actual provincial vote shares, the model correctly predicts the outcome in 273, and makes an error in 32, for a success rate of 89.5%. The model correctly predicted the outcome in all the seats in Alberta, PEI and Newfoundland. There were 2 errors in Nova Scotia, 1 in New Brunswick, 2 in Quebec, 16 in Ontario, 1 Manitoba, 4 in Saskatchewan (by far the weakest performance), and 3 in B.C.
The national total from all this was: L - 137, Cons - 95, NDP - 22, BQ - 54, Other - 0, close the actual results.

I have been doing this modelling for awhile and this was actually one of my better nights, although normally the model is better than 80% accurate.

The model numbers from last week's polls were nowhere near the final seat outcome, but that is principally because a reported average 4% Liberal lead in Ontario became an actual 13% Liberal lead. Not surprisingly, a shift on this scale has a huge impact on the results generated by the forecast model.

What Happened with the Polls

I have finished a full comparison of the polls to the preliminary results and can add a little to yesterday's posts.

There was indeed a shift to the Liberals at the end of the campaign. However, it was confined to Ontario and Quebec. Interestingly, there was a distinct migration from Conservative and NDP intentions in Quebec to the Liberals in addition a shift from the Bloc. The final impact meant that instead of the mid-20's the Liberals wound up at almost 34% in Quebec.

In Ontario when you compare just the last three polls (Reid, Ekos and Léger) you find the Liberals 6% higher on voting day, the Conservatives 3.2% lower and the NDP 2.2% lower. My guess is that many Liberals in Ontario became in effect shaken loose from their traditional loyalties by the combined impact of the McGuinty budget and the sponsorship scandal, but only focused on the federal choice and its implications in the last week. Learning from the polls and the published seat projections (which I think were close to the mark at the time despite all the silly criticism they are getting now), the Liberals in Ontario returned home for the reasons now being cited in the media.

In Quebec, I think the impact of the Landry statement was the difference in the late movement. It came simply from federalists who were upset that the Bloc and PQ would interpret Bloc success as a mandate for a renewed push on sovereignty.

All of the shift at the national level can be explained by the movement in Ontario and Quebec. The shifting around elsewhere is small and appears to me simply within the usual margin of error.

Polls - Post Mortem 2

A brief comment on the polls.

It appears that there was significant opinion movement during the last three days after the final polls right up to voting day, but it was principally in Ontario. I haven't completed a comprehensive review yet but it is clear that in Ontario about 3% of those intending to vote Conservative and about 2% of those intending to vote NDP in Ontario switched to the Liberals making all the difference in this election. Instead of a lead of about 4%, the Liberals ended up with a lead over the Conservatives of 13% with a weaker than anticipated NDP. In other regions I can't find significant movement. There was movement the last week in Quebec, but the Leger polls picked that up.

I heard Stephen Harper tonight blaming his troubles in Ontario entirely on NDP supporters voting against him out of fear. However, he was only partially right and ignored the (slightly) larger phenomenon.

The last week polls elsewhere had no more error than one might ordinarily expect, and the very last poll by Léger in Quebec was quite accurate.

Post Mortem 1

It appears that almost all the difference between the election outcome and the last pre-campaign polls was last minute switching in Ontario including a fair bit of NDP to Liberal strategic voting. This is the first time I can recall such a late shift in Canada.

As for my seat projection model it appears to have worked about as well as one can expect. The large errors in the seat totals stem from the difference between what the polls said would happen in Ontario, and what actually happened.

When I input the preliminary regional vote shares from cbc.ca I get the following outcome nationally: Liberal - 137 (Current actual is 135), Conservative - 95 (Current actual is 99), NDP - 22 (Current actual is 19), and BQ - 54, the same as their current total.

Seat by seat comparison will come later but it looks like, even based on those notoriously inaccurate pre-election day polls, that I correctly identified the winner in every seat in Alberta and all but two in Quebec.

Closing tick

Sometimes one can see the equivalent of the stock market's closing tick in politics. It is definitely there I think in the last few polls out of Quebec that show a trend back to the Liberals. Combined with the silent federalist vote, a well-established phenomenon, it could make for at least a modest disappointment for Mr. Duceppe.

The final polls published in the campaign finished calling people on Thursday night, so any effect from the weekend won't show up until tonight.

I did just hear this afternoon about some fragmentary confidential polling that is a little hard to interpret, but it suggests that the Conservatives may be just a little weaker, and the Liberals a little stronger, than the closing polls reported. And the race is close enough that, if this information reflects a real trend, then the Conservative minority pointed to by the weekend polls may turn out to have been a mirage.

In the end I am just curious to know how it turns out.

Once digested, I will write some analysis of the results and own up to how well (or poorly) the model performed.

A new poll and a revised forecast

There is yet another poll out - a Leger poll distinct from the one released yesterday - that shows a slightly tighter race in Quebec. This suggests that, combined with the fact that pollsters many times in the past couldn't find a hidden federalist vote in Quebec, my Bloc projection below might be still on the high side.

I also discovered a good old fashioned data entry error when reviewing my spreadsheets this morning. My revised numbers are: Liberals - 102, Conservatives - 119, NDP - 33, BQ - 54.

Final Polls Point to Conservative Minority

The final polls are pointing clearly to a weak Conservative minority government.

I have applied a weighted average of the final campaign polls to my forecasting model. The resulting outcome is: Liberal - 103, Cons. - 116, NDP - 33, BQ - 56.

The Conservative vote is more efficiently distributed than the Liberal vote. There is also a slight age bias that helps the Conservatives. They receive a disproportionate share of their support from those over the age of 60 - voters who are much more likely to turn out. I think the Liberals need more than a 1 point lead to pull ahead of the Conservatives. They also need a slightly stronger advantage in Ontario than the 3-5 point lead in the closing polls they have.

Two caveats:

However, a handful a entirely idiosyncratic results in Conservative-Liberal results could easily reverse the outcome above. It could come from the natural advantage enjoyed by incumbents alone.

My calculation of the Ekos numbers puts the Conservatives ahead. They are new to seat prediction, but as owners of the data they ought to be able to forecast seat outcomes more accurately than myself. Perhaps there is some distribution in the data they can see that favours the Liberals that is invisible to someone extracting the regional numbers from a newspaper.

Of course, the Liberals will likely be ahead in the seat count before the B.C. results come in so we will all have to stay up late on Monday to know the final outcome. Who knows? It could our version of Florida 2000. Is the Supreme Court ready?