When one bad poll turns into two front-page stories

Here are the last three AC Nielsen polls:

  • 20-23 April: ALP 51% of the 2PP vote
  • 18-20 May: ALP 54% of the 2PP vote  
  • 15-18 June: ALP 51% of the 2PP vote

And here are the corresponding SMH headlines:

Let me give a more likely explanation. Not since 1977 has a major party has gotten 54% of the 2PP vote. So as I argued in May, we’re probably just seeing statistical noise in the May poll. Indeed, when you take into account the error margins, neither of these movements is particularly statistically significant. (By contrast, as Bryan Palmer recently reported, the betting markets have been rock-steady in the Coalition’s favour.)

Ah, statistical noise – it must be every editor’s dream. So much easier than writing about those boring old policy thingamagies.

Update: More from Mark B.

This entry was posted in Australian Politics. Bookmark the permalink.

23 Responses to When one bad poll turns into two front-page stories

  1. Pingback: Pollitics » The Road to Surfdom

  2. Sacha says:

    Yes, minor movements in a poll does not news make. Betting markets are usually about who people think will win the next election – I think it would be useful if polls asked that question as well each time.

  3. Peter Brent says:

    Andrew, the AEC has several elections since WWII with a 2pp greater than 54%, most recently 1977: http://www.aec.gov.au/_content/When/elections/hor2party.htm

  4. Andrew Leigh says:

    Whoops – thanks Peter. I’ve corrected the post.

  5. Peter Brent says:

    Enough mit the betting markets! They consist of people who read opinion polls and newspaper articles and everything else, and then form opinions. They tend to reflect received wisdom, and when that received wisdom is right, so are they. When it’s wrong, they are too.

    Most people who place bets expect the Coalition to win the next federal election. This is interesting, but has no force when we consider what will actually happen.

    So there.

  6. Andrew Leigh says:

    Peter, you’re viewing the betting markets through an opinion pollster’s lens. No-one has ever claimed that betters are a representative sample of the community. Instead, you should think of them as like futures markets. The people who trade wheat futures on the ASX are not a cross-section of farmers, but nonetheless, their prices accurately tell you the probable wheat price next year.

    But don’t just think of it in the abstract. Instead, you can look at our 2002 and 2005 papers, in which we show that empirically, betting markets are a more accurate predictor of election outcomes than pollsters.

  7. Bring Back EP at LP says:

    Andrew, aren’t you comparing apples with oranges.

    No opinion pollster would say an opinion poll would predict an election except the one taken a few day beforehand.
    The opinion poll taken merely tells us what public opinion is saying at present.

  8. Sacha says:

    Andrew, are you thinking of looking at how the betting markets performed after each election? It’d be a worthy long-term project (hopefully taking not much effort).

  9. Peter Brent says:

    I’ll read those papers. But ‘more accurate predictor’ seems like a broad brush. In 2004, the final Nielsen and Galaxy polls indicated increased majority for government. I don’t know of any betting markets that went into such detail, but would think that that if any did exist, the popular money would have been on a government win with a decreased majority. That was, again, the received wisdom, but it was wrong and those two pollsters were right.

  10. Andrew Leigh says:

    Sacha, that’s what our two papers do. We may also write a post-2007 paper, though there are diminishing marginal returns.

    Homer, there are two answers to your question. First, I’d be happy if pollsters and journalists openly admitted that today’s polls have no predictive power for the 2007 election – but could an editor then justify putting them on page 1? Second, even election-eve polls on average do worse than election-eve betting markets.

  11. Corin says:

    Andrew,

    Is there any research on the difference bewteen when the poll “matters” or not. I mean it is possible that post Budget the polls could change that much – or is a complete statistical aberration?

    I mean I wonder say with Latham’s 4-6 months of consistent 54 tpp – whether it was a “real” vote level or an indicator that people were “wanting” to vote for him – but still may not and werelikely to switch back. i.e. When you don’t have to wake up with him as PM tomorrow its’easier to say I’m voting Latham.

    I also think the preference shown by many for Gillard in polling has the same ring to it.

    Anyway – I think the party’s have always favoured qualitative results this far out from an election. I’d say but would love to see the real qualitative polling that the government “narrative” is still dominant with u/e at 4.9% and looking to go lower.

    I’d also suggest that a lot of people will be saying: “when Beazley is the best they’ve got – I’m not sure they have the depth”; or something like that.

    Have you seen the qualitative polling??

    Corin

  12. Ben Ticehurst says:

    Perhaps polling makes us lazy. Humans are very social animals and can act like herds at times. Perhaps the widespread reporting and emphasis on polls serves to magnify positive and negative feedback spirals in voting patterns much like the instabilities in stock market booms and crashes.

    Do polls have a useful purpose for the public? (Obviously private polling benefits parties.) We might be forced to think and choose carefully if we weren’t sure how Joe next door was voting.

  13. Sacha says:

    Andrew, yes I meant after the 2007 election and maybe some further on down the track (despite the diminishing returns) as might could give it more oomph.

  14. I don’t know. I go back and forth on this. Newspapers are into selling papers, day in, day out. We’re not. There is a fundamental divergence of interests here. So, on the one hand, I’ve stopped being outraged when they try to spin a 3 point wriggle on n=1,400 or so into something meaningful (thats what a daily newspaper does or has to do); on the other hand, if we don’t stop being outraged (and telling them about it), will they ever stop? Notice that they lead with the poll result (ALP vote share down!), and not “betting marketes show no change”.

    Re Andrew and Peter Brent on the betting markets. Some unresolved questions lingering there. If the markets are so good, why are they so cautious? That is, when the government wins in 2004 with an increased majority and 52.74% of the 2PP, I believe the closing price in the betting markets was something like a .77 probability that the government would win. But in any event, it is way more conservative than if you conditioned on the final polling numbers alone: the last big-4 media polls before the 2004 election were

    org start end mode sampleSize govt var
    8 AC Nielsen 5-Oct 7-Oct telephone 2029 0.54 0.0001224248
    15 Galaxy 5-Oct 6-Oct telephone 1200 0.52 0.0002080000
    27 Newspoll 6-Oct 7-Oct telephone 2500 0.50 0.0001000000
    39 Roy Morgan 7-Oct 8-Oct telephone 1311 0.49 0.0001906178

    the precision weighted average of those polls is .5131, the std err on that is 0.00595, and via the usual normal approximation, the probability that the point estimate (the precision-weighted average) exceeds .5 is .986. Note that under uniform swing, the Coalition could still have won with 49.7% of the 2PP, and so the poll-based estimate of a Coalition win was approximately 1.0.

    So, who made a better prediction re the election?

  15. Andrew Leigh says:

    Simon, I love your aggregation paper (though I seem to recall that it also included the wonky ANU-Bulletin number, which skewed things somewhat). But when we do a similar poll aggregation exercise over the period before the election, we get wild swings – 0.6% to 99.9%, from memory. I just don’t find it credible that the underlying attitudes are as volatile as even the aggregated poll number implies.

  16. Sacha says:

    Simon – just a tiny point – empirically there are no such things as uniform swings – they don’t exist.

  17. Sacha says:

    Also, it doesn’t make much sense to say that a party could win with a certain 2 party preferred vote, as the votes for a party can be arranged across the seats in all sorts of different ways to construct a win on practically any reasonable 2PP vote, eg look at the 2001 election, or the 1990 election.

  18. Peter Brent says:

    My understanding is it’s a bit misleading to say that a 51 percent poll result is really 51 + or – three percent (or whatever the number is) and that therefore the number is really ‘anywhere’ between 48 and 54. Isn’t it the case that the number is much more likely to be in the middle than on the outer fringes, for example it’s more likely to be between 50.5 and 51.5 than between 53 and 54? if you know what I mean.

    So if, rather than using 95 percent interval, we said ‘on the balance of probabilities’, and used only, say, 51% confidence interval, we’d get a much smaller range, and could say ‘that poll showed the number would ‘probably’ be between [a much smaller range, around 51 percent]

    does that make sense? If so, does anyone have the formula that allows you to insert whichever confidence interval you want to use?

  19. Andrew Leigh says:

    Peter, here’s a calculator that does what you want. As always, Wikipedia is a great resource.

    In my last post, I did something a smidgin trickier, which was to use the “prtesti” function in Stata to find out whether the difference between two polls was statistically significant. The answer is that with 1400 respondents, the difference between 51% and 54% is only significant at the 11% level (aka the 89% level).

  20. Geoff R says:

    My intuitive understanding of the argument for the use of betting markets is that they reflect a judgment of how voters will behave in the future. For example the very large Liberal leads early in the Tampa saga reflected voter approval of the government’s position on the dominant issue in the news, but when citizens actually voted they took over issues into account. It would have been foolish to put money on an 80 seat Coalition majority even if the polls suggested this. But how reliable would betting markets be without public opinion polls? We don’t know because they did not exist. The only public representation of opinion is the press but 1930s political journalism is mostly so partisan that it would have been of no use for betting markets. The SMH’s marginal seat predictions in 1929 were mostly wrong. Perhaps the strategies of (vote-maximising?) politicians provide evidence of what public opinion was.

  21. Sacha; you won’t get an argument from me on uniform swing (I’ve been fighting that fight longer than most of you); my point was to say that the Coalition probably could have won with slightly less than 50% of the 2PP vote, as it did in 1998 (as could Labor, as it did under Hawke), such that 50-50 was probably too-conservative a breakpoint for when to map aggregate 2PP into a win/loss for the Coalition.

    Andrew: you are of course right to note that the polls are over-dispersed (i.e., they display more variation than they should given their nominal sample sizes), especially on 2PP, due to house effects. Hence that AusJPS piece of mine, trying to find better ways to use the information in the polls.

    I reported what statisticians call power calculations in my AusJPS piece on the polls in 2004. Its a joke. To get the conventional 95% significance level so as to detect a real bump from 50 to 51 support, you need over 50,000 respondents per survey. Go fish.

    yes, the ANU on-line poll was way off in 2004. If you included it real-time aggregation exercises of the polls in 2004, it messed you (um, me!) up. but if you took the three phone polls (Nielsen, Newspoll and Galaxy) from about 4 or 5 months out, I don’t think you ever had anything other than a Coalition win with prob > .5; and indeed, from about the 2nd week of the campaign, if you simply aggregated just those 3 polls (no ex post bias corrections), I think you’d see that you always had the Coalition 2pp above 50% with prob > .95.

    Geoff R is spot-on. The counter-factual of a betting market in the absence of opinion polling would be fun… we basically get those in the marginal seats, the seat-by-seat markets, kinda…

    and finally, Peter B is right: its not that when we say a margin of error we mean that our beliefs are “that the truth could be anywhere in that margin”. Absent a bias term for a poll’s house effect, you should imagine a bell-shaped, normal distribution centered on the poll’s estimate; since its a normal distribution, you’ll recall that 95% of the area under that bell-shaped curve lies within plus/minus 2 standard deviations of the mean; two facts: (1) the standard deviation is going down at rate proportional to the square root of the sample size, hence the colossal sample sizes needed to reliably detect small movements at conventional levels of statistcial significance; (2) the more risk of being wrong we’re prepared to accept, the smaller the margin-of-error decreases non-linearly (e.g., a poll of 1,400 finds a 50-50 split, the conventional 95% margin of error is +/ 2.6 percentage points, but a 50% margin of error is +/ 0.9 percentage points (i.e., you’re just as likely to find the true number in that considerably smaller interval as not).

  22. Peter Brent says:

    Thanks Andrew. But alas, it only goes down to 80%

  23. Pingback: Andrew Leigh » Blog Archive » Poll ca change

Comments are closed.