Skip to main content

What good are polls except to fill the gogglebox?

The Wilson Quarterly surveys polls.

Despite unanimous poll results predicting an Obama victory in the New Hampshire democratic Primary, pollsters still determined Hillary Clinton was going to emerge the ­winner.

The New Hampshire debacle joined a list of major embarrassments that includes the disastrous Florida exit polling in the 2000 presidential election, which prompted several networks to project an Al Gore victory, and the national polls in the 1948 race, which led to perhaps the most famous headline in U.S. political history: “Dewey Defeats Truman.” After intense criticism for previous failures and equally intense efforts by pollsters to improve their techniques, this was not supposed to ­happen.

New Hampshire gave new life to many nagging doubts about polling and criticisms of its role in American politics. Are polls really accurate? Can surveys of small groups of people give a true reading of what a much larger group thinks? What about bias? Don’t pollsters stack the ­deck?

Iowahawk also chimes in on the margin of error of what is a very human process:

As a rule of thumb the margin of error of a sampled probability is:

Margin of Error = 1 / sqrt(n)

So if the sample size is 400, the margin of error is 1/20 = 5%... if the sample size is 1000, it's about 3%.

Obviously this works well for controlled experiments, defective widgets, jellybeans, and coloured balls in a jar. It's also good for undergrad exam questions. But what if the things you are studying don't quite fit the curriculum of QUAN101, what if the subjects are more wormy... say, like voters?

Using the analogy of coloured balls in an urn:
  • What if 40% of the balls have personally chosen to live in an urn that you legally can't stick your hand into?
  • What if 50% of the balls who live in the legal urn explicitly refuse to let you select them?
  • What if the balls inside the urn are constantly interacting and talking and arguing with each other, and can decide to change their color on a whim?
  • What if you have to rely on the balls to report their own color, and some unknown number are probably lying to you?
  • What if you've been hired to count balls by a company who has endorsed blue as their favorite color?
  • What if you have outsourced the urn-ball counting to part-time temp balls, most of whom happen to be blue?
  • What if the balls inside the urn are listening to you counting out there, and it affects whether they want to be counted, and/or which color they want to be?

If one or more of the above statements are true, then the formula for margin of error simplifies to

Margin of Error = Who the hell knows?


Related Link:

Comments

  1. There's an interesting post up on Holy Smoke on this - mainly on the perspective that the BEEB was preparing for a Kerry victory last US election, and how it affected their morale negatively when Bush won. And some of the comments indicate that the polls in the US are not to be trusted even though the poster thinks that Obama's going to win.

    ReplyDelete
  2. Greg, those surveyed are not the problem, whether they be widgets, professors or chimney sweeps. Valid survey design ensures the answers & who they came from can be seen & understood, even if this doesn't tell us the answer we wanted to know ;)

    The real danger is not so much dodgy analysis of survey results (as these are often disclosed to an extent that dodgy technique is exposed), nor the collection of the survey answers (which is usually done by underpaid students with little interest in politics), but in the design of the survey questions. Skew the survey, and the answers will definitely be skewed...

    Which is why survey questions should almost always be revealed, as should non-response rate (those "50% of the balls who live in the legal urn [who] explicitly refuse to let you select them").

    It is teeth-grindingly frustrating to see unscientific self-selected online surveys (hands up, Herald and Stuff) quoted as authoritive sources of public opinion. Especially given Kiwiblog reports of deliberate stacking of results.

    But the classic NZ political poll gaff/skew is to not bother the punter by listing all possible political options. For example, asking who is preferred PM and only offering a choice of Key vs Clark - kinda explains why no other party leader troubles that poll much eh? The respondent has to actively push for their choice to be recorded...

    Had a UMR poll like that the other day - clearly contracted for the blue balls in the urn ;)

    ReplyDelete
  3. Have you been "push-polled"? It's hilarious.
    ACT phoned me 2 weeks ago and by mistake again last week.
    The theme was emissions trading.

    All questions were framed, "Did you know XYZ?" To which the only answer was "why yes I too hold the same biased opinion" or "really is that a fact, no I didn't know, how incredibly interesting".

    I don't know whether the poll was done before and gave influence to, or done in concert with the ACT ETS billboards. Doubtlessly it's coordinated.

    ReplyDelete
  4. Hi Greg,
    Nope, haven't been push polled, tho it sounds like fun ;) 10 minutes of only 1 way to answer! Kinda like a North Korean pollster asking you if you prefer the Dear Leader or Precious Leader more (cue firing squad warming up in the background)...

    Either ACT are very desperate, or need to do some real polling - they have put up 'dump the ETS scam' billboards in my very working class neighbourhood (decile 1-2), which have promptly been tagged. It looks like the locals have 'zero tolerance for Roger Douglas'...hehe

    What is frustrating in the election is lack of polling in individual electorates. I don't really expect the media or parties to pay for that level of detail, but it would be nice if the state funded Elections NZ to do such polling. Ya can wish anyway...

    ReplyDelete

Post a Comment

Please be respectful. Foul language and personal attacks may get your comment deleted without warning. Contact us if your comment doesn't appear - the spam filter may have grabbed it.