Forecast Model Errors

Every election I test the accuracy of my forecasting model by using the actual vote shares as inputs. In other words this exercise treats the provincial vote shares as if they were the perfect opinion poll. This isolates the model’s error from any polling errors. I don’t use the vote shares for each province. I group the Atlantic Provinces into one region and make Manitoba/Saskatchewan as another region, exactly as most national opinion polls do.

The results I measure in detail in two ways. First, did it accurately predict which party would win individual constituencies? Out of the 308 ridings the model accurately predicts the outcome in 279 ridings, and incorrectly in the remaining 29 for 90.6% accuracy.

Secondly I also compare each party’s predicted share with its actual result, turning the pluses and minuses into absolute numbers, and then I average the errors.

The average error by party was:

Liberal – 3.4%
Conservative – 3.2%
NDP – 2.2%
BQ – 2.6%
Green – 0.8%
Other – 1.4%

On the whole this compares favourably with earlier elections. The average error per seat in 2004 was slightly higher. With the two elections closer together this is perhaps not surprising. However, before the election I had thought seat predictions might be less accurate. I was wrong about that.

It's the economy, stupid...

The economy recently has been booming along so well that this past election was an argument about how to spend the fiscal surplus. Stephen Harper comes to power at a bad time in the sense that, even though the unemployment rate is at a thirty year low and the stock market is hitting new records, he has to hope it is sustained over the next eighteen months since slippage will be blamed fairly or otherwise on him.

So could a recent dramatic slowdown in U.S. GDP growth mean that we are on the verge of a slowing economy, which will pull down both Stephen and George?

Effect of election on same sex marriage

Not surprisingly, there are not enough votes out of Monday's election to overturn same sex marriage legislation. Bouquetsofgray has the details. This is actually good news for Stephen Harper.

Not so this story in today's Globe that Harper ducked his obligation to meet with the ethics commissioner about his role in the Grewal affair. Didn't take long.

A close election

Various observations I would make are:

1. The election did end up being closer than I anticipated. So close indeed that if the Liberals had been two points higher in Ontario and one in B.C. while the Conservatives were correspondingly lower, the outcome would have been reversed even if all other provinces had voted exactly the same way. Overall, had this been the case, the Liberals would have been four points down in the popular vote but ahead (barely) in seats. I was able to do this exercise because I have already built my model for the next election, courtesy of some preliminary data files on the Elections Canada web site. I simply varied the outcome from yesterday just enough to produce a Liberal win.

2. The closeness makes me think that if the campaign accidents such as the income trust affair, and certain stumbles, such as the military ad, had not occurred, this campaign could easily have had a different outcome. Certainly, the Liberal campaign planners made copious errors, while having to cope with an unfriendly media. As one example of the latter, I couldn’t believe the blatant shilling for the Conservatives I sometimes saw on the Mike Duffy show. As one friend noted, at least he wasn’t subtle about it.

3. My forecast model worked reasonably well, but not as well as in 2004 (as I anticipated here). I test the model’s accuracy by using actual vote shares (which are the perfect opinion poll). The results generated were:

C – 124
L – 94
NDP – 31
BQ – 59

The Liberal number, as you can see, did run ahead of the trend and demonstrated great vote efficiency. And Quebec contributed disproportionately to the number of errors.

But I have more work ahead to complete a full evaluation. I also want to do a full evaluation of poll accuracy. As many have noted, SES appears to have been the most accurate. Once again the Quebec pollsters were ahead of their national counterparts in getting Quebec public opinion right, although SES was also close here.

When I apply the average of closing polls to my new forecast model based on yesterday’s results, I get the following:

C – 132
L – 91
NDP – 36
BQ – 48

The largest difference, as it was last time, was in Ontario.

4. In some ways the Liberals will benefit from being out of office, but not far removed from power. This defeat will serve as punishment for the scandal and should end its political impact. Although this may be delayed until the trials are over in the spring.

5. The key for the Liberals is the party leadership. And here it is not at all clear who among all the potential candidates would be the best choice.

6. This election also completes the NDP’s full resurrection from the political graveyard they hung around throughout the nineties. They have picked up some interesting new talent, and have an energetic leader who is becoming progressively better in the role.

9. All elections do generate some clichés in the post-election analysis. One, here on the CBC internet site, discusses an urban-rural divide. While the Conservatives did very poorly in the three biggest cities of Toronto, Montreal & Vancouver, they finished first in all the prairie cities including Winnipeg, as well as Ottawa, Quebec City and St. John’s (see Major Centres results here), so I think some qualifiers are in order.

Assessment

I will provide an assessment of the results tonight. The headline is that the major influence on the outcome that differed from the pre-election poll averages was greater Liberal strength than anticipated in Ontario. However, there was movement in that direction at the end. SES at first glance seems to have done the best job of picking it up. More later.

These are really the final numbers.

I have now incorporated the final Strategic Counsel and a final Ipsos-Reid poll (done partly online) into the averages and applied them to the seat forecaster in two ways. In the first case I use only the Quebec polling firms for the Quebec numbers (in previous elections Quebec-based polling firms have been much closer the actual result than the national firms); in the second I have used the national pollsters Quebec sub-samples.

With the Quebec polling firm numbers for Quebec
C – 134
L – 77
NDP – 41
BQ – 56

With the national polling firm numbers for Quebec
C – 129
L – 75
NDP – 41
BQ – 63

Here is what Strategic Counsel on its own gives us:
C – 126
L – 75
NDP – 45
BQ – 62

Here is what Ipsos-Reid on its own gives us:
C – 129
L – 74
NDP – 42
BQ – 62

There you have it. Now I am off to watch.

New Poll from Strategic Counsel

We finally have the last poll of the campaign. One came in last night after I posted from Strategic Counsel. I will incorporate it into my averaging and post a final, final set of numbers early this evening.