Online polls

Of the two polls referred to in my February 15 post, one, Angus Reid, was an online poll (methodology outlined on the first page) while the Ekos poll (methodology on the last page) was conducted by an innovative telephone technology, interactive voice recognition or IVR for short, essentially a recorded voice that asks the questions while respondents as Ekos states "enter their preferences by punching the keypad on their phone."

Increasingly the polls being reported in the media are conducted online.  Essentially, firms recruit a very large panel of respondents by advertising on the net and getting voluntary sign-ups. With the panel's email addresses the pollster then conducts a poll by randomly selecting members of the panel to receive answer an internet survey.  However, the methodology is new and essentially experimental. Certainly there are no clearly defined agreed upon standards for online polls.

Recently, Nate Silver of the New York Times wrote a long and intelligent blog post entitled Before Citing a Poll, Read the Fine Print, about this new methodology:

Internet-based polls are very likely to be a part of polling’s future, and my view is not necessarily that they should be dismissed out of hand. However, they need to be approached with caution.

The central challenge that Internet polls face is in collecting a random sample, which is the sine qua non of a scientific survey. There is no centralized database of e-mail addresses, nor any other method to “ping” someone at random to invite them to participate in an online poll. Many people have several e-mail addresses, while about 20 percent of Americans still do not go online at all.

The situation can be contrasted with the platonic ideal of a telephone poll, in which everybody has a phone number, and they each have an equal chance of being reached through a random digit dial method.

In reality, telephone polling falls short of the platonic ideal, while the best online polls take steps to make their samples effectively random. Some telephone polls, especially those that are conducted through automated scripts, do not call cellphone numbers, even though more than a quarter of American households do not have landline telephones at all, with the fraction increasing by several percentage points every year. Meanwhile, households often share a single number between a family, or they have multiple telephone lines; careful pollsters take steps to ensure that their samples are not biased by these problems, but others apply a blitzkrieg approach to polling and do not.

Practices for conducting online polls vary significantly from survey firm to survey firm.....

My view is that online polls should be regarded as “guilty until proven innocent.”

TC completely endorses the latter sentiment.

If one looks at Canadian polling firms one finds a full range of methodologies.  There are some including Nanos Research and Ipsos that use traditional telephone methods of conducting a survey. Angus Reid uses online surveys from its panel, volunteers recruited from the internet. Forum Research mainly uses interactive voice recognition for its media polls but on its web site suggests that it uses a variety of research methods.  Ekos research used interactive voice recognition for its election polls in 2011 but also deploys a combined methodology it describes as a "unique, hybrid online/traditional random digit dialling (RDD) panel" (see p. 12). It also notes that all "respondents to our panel are recruited by telephone using random digit dialling and are confirmed by live interviewers".  Abacus polls online as in this latest federal party preference survey.

A recent example illustrates why we need to be skeptical about all polling.

Nanos released a poll on March 15th on Ontario party preference.  It described its methodology this way:

Between March 3rd and 5th, 2012, Nanos Research conducted a random telephone survey of 500 Ontarians 18 years and older. A random telephone survey of 500 Ontarians is accurate plus or minus 4.4 percentage points, 19 times out of 20.  For 428 decided voters, the margin of error is accurate plus or minus 4.8 percentage points, 19 times out of 20.

The poll reported that the Ontario Liberals led with 39.9%, the PCs were second with 30.0% while the NDP had 24.7%.

This day before Forum Research released a poll with this methodology (as described in the Toronto Star):

Forum’s interactive voice-response telephone poll of 1,065 people, conducted on Tuesday, is considered accurate to within 3 percentage points, 19 times out of 20.

However, the poll had the PCs leading with 40%, the Liberals at 28% and the NDP at 23%. 

It was two weeks later than Nanos but the time lapse cannot possibly explain the different result.  It is a lesson that the results of a single poll can be dead wrong. Which one of these two is wrong?  Who knows? And we can`t find out as there is no imminent Ontario election on the horizon to confirm one outcome or the other.