Reporting on noise

From Mark at Larvartus Prodeo:

As Christian Kerr wrote yesterday, the important thing to watch in the polls is the trend, not individual surveys.

No. Definitely not. If polls are measured with a lot of error (and we know they are), then the difference between two polls will be measured with even more error. The upshot is that a news outlet that tells us “Labor is down 2% in the polls this week” is about as useful as a weather report that says “the chance of rain today is 10% higher than yesterday”. 

Here’s Simon Jackman on the topic:

Two polls with sample sizes of about 1,500 each have about a 50-50 shot at finding statistically significant changes when the real change is 4 percentage points, and it only gets worse for smaller changes in vote share. By the way, four percentage points is a lot of change, relative to the variation in actual vote shares in Australian electoral history.

My last post on the topic.

This entry was posted in Australian Politics. Bookmark the permalink.

9 Responses to Reporting on noise

  1. bryan says:

    Andrew, I think Crikey and Mumble meant “trend” in the non-statistical sense — as in moving average or trend line over a number of polls. Their underlying point, with which I am sure you would agree, is don’t read too much into a single poll result.

  2. Yes, that’s what I took Christian to mean.

  3. Pingback: Club Troppo » The tortoise strategy

  4. It’s curious that after Morgan and Galaxy both showing a similar movement no-one wants to suggest that the Therese Rein affair had a negative impact on Labor’s support.

  5. Andrew Leigh says:

    I’m happy to take Mark and Christian at face value. But ‘moving average’ is neither the technical nor layman’s definition of trend. As Don Arthur has pointed out, their use of ‘trend’ is a lot like The Australian’s use of the term, which is focusing on changes, rather than moving averages.

  6. I think just like the Morgan polls, that showed the same result before Galaxy but were ignored, people will interpret polls in a way that suits them.

  7. Verdurous says:

    I suspect polling creates dangerous positive feedbacks with respect to voting intention. In a similar way, I think it was unhelpful for the media to declare a “winner” in the democratic presidential candidate debate this week in the US. (Unfortunately, people seem to often decide what to do based not on the content of what is said, but on what other people around them are thinking/doing. A little bit like the positive feedbacks that take hold in stock markets, creating bull or bear runs. Irrationality prevails. Any thoughts? Is polling useful, and if so do the benefits outway the problems? Does polling contribute to an abdication of one’s responsibility to think for onesself?

  8. Peter Fyfe says:

    It would appear there is even more noise in reading a poll than there is in the underlying data.

    I’ve never heard an average I found particularly moving. Sorry 🙂

  9. Pollio says:

    I think Bob McMullan’s article in the Oz last week is an interesting contribution when talking about ‘trends’ –,20867,21803293-7583,00.html

    Imagine a trend line which goes up four points then down 2, up 4 then down 2 and so on. The ‘trend’ is continually upwards but the desparation of the papers for movement from one poll to the next instead reports votes that are bouncing up and down.

Comments are closed.