Methodological Snake Oil

TC wants to make some points about the polling controversy started by the Ipsos commentary about polling discussed in my previous post.

First, there are legitimate unanswered questions about online polls.  There is no accepted methodology.  Partly because of that there is effectively zero transparency and accountability on the part of online polling companies.  For example, on their website at the bottom of a recent poll the Reid group says this:

Methodology: From September 20 to September 21, 2011, Angus Reid Public Opinion conducted an online survey among 1,668 randomly selected Canadian adults who are Angus Reid Forum panellists. The margin of error—which measures sampling variability—is +/- 2.4%, 19 times out of 20. The results have been statistically weighted according to the most current education, age, gender and region Census data to ensure a sample representative of the entire adult population of Canada. Discrepancies in or between totals are due to rounding.

What it doesn't tell you is how large the panel is from which the sample is selected. When it says 'randomly selected' it goes on to say 'who are Angus Reid Forum panelists'. However, the phrase 'randomly selected' on its own implies 'selected from the whole population of Canada'. The subsequent reference to the panel also infers, without saying so explicitly, that they actually come from the Angus Reid panel.

In Canada if it is within a province, one assumes it is a subset of a national panel and perhaps panel size within the province could be an issue. But we don't know the overall size or demographics of the national panel - gender, income, language - let alone the provincial sub-sample. We don't know what methods were used to recruit it. Surely if we are to accept the panel itself as representative we would need to be assured that it was selected at random. Is this case?

We don't know how the panel is contacted or how the surveys are conducted to ensure their integrity. If these are to become commonplace, the polling industry should set some standards that provide some reassurance that there is some validity to all this.

TC doesn't agree with the Ipsos critique of IVR.  These are ultimately phone surveys, an established proven methodology.  The only difference is an electronic voice and answers given on a telephone touchpad.  We already know such polls have obtained good results. See what New York Times polling blogger Nate Silver (who formerly blogged at fivethirtyeight.com) said about Survey USA, which uses this methodology:

... SurveyUSA is a very strong polling firm; no company has done more to contradict the notion that a "robopollster" need be inferior. Although it's not my place to make any endorsements, it would certainly make the life of electoral forecasters easier if SurveyUSA were to get more business.

Competing Ontario polls

Two new polls out in Ontario today ahead of the debate, one with the Liberals ahead, one with the PCs, so which to believe.

An Ekos poll had the Liberals at 34.9%, the PCs at 31.4% with the NDP at 24.7%.  However, Abacus Data had the PCs ahead at 37%, the Liberals at 33% and the NDP at 23%.  The only big difference is actually in the PC number but it is significant.

One way of verifying the credibility of a poll is to look at its internal subgroups.  There is a certain consistent pattern that reappears in poll after poll regardless of overall outcome or source.  If there is an anomaly it should be apparent.  Look on page four of the Abacus poll.  It tells us that the PCs lead among women and voters age 18 to 29.  So younger voters and women support the PCs almost to the same degree as the province as a whole.  However, there is absolutely no way that is true.  PC support is typically older and male to a much greater degree than we see in Abacus.  In the Ekos poll (see page 7) one sees the relative weakness of the PCs among youth and women, and one sees it over and over again in many others.

Abacus was attacked by Ipsos Research (the most accurate pollster in the May 2 federal election) for shoddy methodology (methodological snake oil is how they described it) earlier in this campaign. Abacus made a point in their release of emphasizing that they used a traditional ballot question on this poll (in response to the Ipsos critique).  TC thinks Abacus has more work to do.

Manitoba polls point in the direction of an NDP win

Two polls with conflicting results were released in Manitoba today.  One was a phone poll by Viewpoints Research for CJOB, which reported 41% for the NDP, 32% for the PCs and 5% for the Liberals with 19% undecided.  This gives the NDP about 51% of the decided vote with the Tories at 40%.

By contrast another survey, an online poll by Environics, found the PCs in a narrow lead 45% to 42% over the NDP with the Liberals at 10%.  The clashing methodologies should be noted. Online polls have had some  real problems, so there is a need to be cautious in interpreting the results. UPDATE: TC goofed - the reference I made previously to real success in the U.S. was to interactive voice recognition polls.

Taken together, however, this polling data as well as earlier polls point in the direction of an NDP victory with a reduced majority.  Even in the Environics poll Selinger has a higher approval rating than opponent Huge McFadyen (54% approval compared to 44%). It must also be remembered that the NDP has a more efficiently distributed vote. Large majorities in the rural southwest "waste" more votes for the Conservatives than comparable NDP margins in the city, and the NDP controls the small population northern ridings.  TC estimates that the PCs likely need a four to five point lead to be certain of winning.

Liberal support is so weak that the party may well face a shutout (as TC noted on Saturday).  One can see on this map of the individual polls from the 2007 election results imposed on the new electoral boundaries that the Liberal polls in northwest Winnipeg are now split between Tyndall Park and the Maples, which should deliver both ridings to the NDP.  If Jon Gerrard loses River Heights that would give us a majority government for certain (even if the margin is narrow).

This is the last week of the campaign and we should still see polls from Angus Reid and Probe Research so there will be more information to consider before voting day.

Ontario and Manitoba elections - no obvious winner in sight.

The first thing that needs to be said about these two elections is that by the usual rules of politics both incumbents should have been heading for defeat from the outset.

In the case of Ontario the Liberals are endeavouring to win a third term with a leader with low approval ratings, a year after introducing an unpopular new tax, the HST, that applies to many more goods and services than the old provincial sales tax.  Third terms are highly unusual, while the party that wins the election usually has a well regarded leader, so the Liberals started out with three strikes against them.

In Manitoba the NDP has a new leader in Greg Selinger who, despite his considerable virtues, is less popular than predecessor Gary Doer (however, he has a much higher approval rating than Dalton McGuinty).  And the NDP is trying to win a fourth term, something that has not been done in Manitoba since Duff Roblin.  Winning multiple terms is difficult in most places simply because all administrations eventually accumulate grievances directed at them from various sectors of the public.

There is less than two weeks to go in both campaigns. The clear narrative for both is that they are close but both incumbents could win re-election (in Ontario's case likely as a minority government).

Ontario
A large sample poll in the Toronto Star today makes it clear that the race is extremely close (similar findings appeared in other polls last week).  The Star poll reports that the survey would result in 47 ridings each for the Conservatives and Liberals and 13 for the NDP. The poll is large enough to have results for every riding.  However, some of the results reported don't look right to TC. My forecast model suggests that a tie vote would produce an advantage for the Liberals as they have had the more efficiently distributed vote in the past. In addition TC would expect to see more seats for the NDP than just 13. But it is tight enough to go the either way and many individual riding contests must be very close.

Despite the Liberals' handicaps it appears that attacks on PC leader Tim Hudak (see, for example, this web site: hazardoushudak.ca) have taken their toll. This Ipsos poll from earlier this month reported that voters preferred McGuinty to Hudak as premier.  The disapproval of McGuinty found in the survey noted above suggests that it may be a case of who is the least preferred. 

Both Tim Hudak and NDP leader Andrea Horvath were not well known coming into the campaign.  The NDP has been in the range of 23 to 26 per cent support, well above the 16.8 per cent the NDP won in 2007.  The NDP's poll numbers this summer look like a spillover from the federal campaign and the outpouring of emotion for Jack Layton, so Horvath still must close the deal with voters to do well. However, this Ipsos poll released a few days ago suggests she is making progress.  The poll also makes it clear that Rob Ford has become a huge liability for the PCs in Toronto. His unpopularity could cost them dearly in this election:

A majority (53%) of 1,719 Torontonians polled say Rob Ford being mayor of Toronto makes them less likely to vote for the Progressive Conservatives, while just 9% say they’re more likely to vote for the PC Party

The television debate coming Sept. 27 could make a significant difference. It represents an enormous challenge and opportunity for both Hudak and Horvath.  I suspect there is less on the line for McGuinty but he still must perform well given the tight race.

Manitoba 
Two significant developments yesterday: a poll reporting that a majority think the NDP deserve re-election and a reasonably successful television debate appearance for Greg Selinger.

The most memorable aspect of the debate was Selinger successfully forcing Liberal leader Jon Gerrard to state that he may have had made "a mistake" in voting against a Selinger budget.  Even though PC leader Hugh McFadyen was on the sidelines for this exchange, it was quite important as the role of the Liberal vote in Manitoba elections is critical given that many federal Liberals support the NDP  provincially.

There is some evidence that Liberals are in deep trouble in this election, and the beleaguered party could lose both its seats. Former MLA Kevin Lamoureux has moved to the House of Commons, and his seat has been split in two by redistribution (both are likely to go NDP) while the PCs are making a strong effort to unseat Gerrard in River Heights.

TC thinks the PC's biggest problem may be that they start out so far behind in the City of Winnipeg where they only held four seats entering the campaign.  In 1995 when the party last won an election they won 14 seats in the city.  The PCs recognize this. The Free Press had an article on the PC effort to win the south end riding of Seine River.  TC's view is that the PC's must win that riding and several others in the city (such as Kirkfield Park, Southdale and St. Norbert plus others) to have a chance of winning.

Nevetheless, the PCs have all the advantages that accrue to being able to campaign on "time for a change", and a leader who is showing more poise in his second campaign.  The election outcome remains uncertain.

Ontario polls point to Conservative minority government

The two surveys out this week provide more evidence that a close outcome is a distinct possibility and that means minority government.

The Angus Reid poll points to a thin Conservative majority of 56 seats. However, just one more point for the Liberals would turn that into a minority. The Forum Research poll would give the Cs 52 seats while 54 seats are needed for a bare majority.

The NDP was strong in both surveys and is currently running well ahead of their pace in 2007.