Andrew Leigh has a lot to answer for

I just noticed (via A Roll of the Dice) this comment from Mumble*:

February 9 Betting markets and the genius of punters
Centrebet is paying $1.90 for a Labor win and $1.80 for a Coalition one. These (I think) are shortest Labor odds (ie longest Coalition ones) since around September/October 2001. Andrew Leigh has a lot to answer for. This “the betting public is better at predicting election results than opinion polls!” shtick has become a standard roll-out for journos and commentators; SMH today two pieces on or alluding to it. (See also Simon Jackman’s take.) You know, the punters, who had the Coalition at over $3.00 in early 2001, but changed their minds when the government took the lead in opinion polls later in the year. Not too bright in early 2001 (or for most of the 2005 WA campaign), were they? Or the dills who on New Zealand’s election day in 2005 had the Nationals winning? All these positions were based, mainly, on whatever the polls said at the time. The punters are sheep who reflect the current received wisdom, which is usually correct, but sometimes wrong. B-a-a-a-a-h. I’ll follow my own judgement, thanks.

Justin Wolfers and I have two papers now in which we show that in Australia, the betting markets are better predictors of election outcomes than the polls. Internationally, others have published maybe two dozen papers showing the same thing. The betting favourite doesn’t always win, particularly when the two parties are close (the answer to the NZ 2005 example is simple: a party with a 45% chance still wins about half the time). But I am aware of no paper that finds that the polls outperform the betting markets as a prediction tool (which is what you’d find if betting markets just followed polls). 

Moreover, experts do even worse than the polls. The Sunday prior to the 2004 election, the Sunday Tele surveyed 10 election experts, and found that three thought Latham would win, while seven thought Howard would win, but with a smaller majority than in 2001.  None forecast the true result – a Howard victory with an increased majority.

Given what I regard as overwhelming academic evidence in favour of betting markets as a prediction tool, perhaps I should turn the question back on those who think that newspapers should give more prominence to polls and expert forecasts than betting odds. Is your position based on faith, or is there evidence beyond what we’ve so far produced that would persuade you to change your mind?

* I would’ve made this point on Peter’s blog if it allowed comments.

This entry was posted in Australian Politics. Bookmark the permalink.

23 Responses to Andrew Leigh has a lot to answer for

  1. If it’s an empirical observation that the betting markets predict better than polls, then that’s fine.

    But I’m very sceptical of the reasons advanced as to why it might be the case. I should note that I haven’t read any of the papers, and I’m just going on what people have reported as the general thesis, so I’m happy to be corrected.

    It seems to be the argument that “insiders” will have a better perception of likely outcomes and that this will be reflected in the betting markets.

    However, this seems to me to be problematic for a number of reasons.

    (1) There’s no evidence that those who bet on elections are in fact “insiders”.

    (2) “Insiders” don’t necessarily have any more access to data than anyone else. And either their information is recursive – ie they’ve seen party polling that outsiders haven’t or it’s in some way intuitive. With regard to the former case, this is very closely held and most “insiders” would know as little about it as pundits or the public. In the latter case, there’s no reason why anyone with good insight couldn’t reach the same conclusion as “insiders”.

    (3) To give an example, in both the Queensland and Victorian state elections, Centrebet offered punters a chance to bet on how many seats Labor would win. The odds in Qld were shortest for the numbers predicted by the pundits (a range of 50-52) while the odds for 59 (the result) were quite long. Either the money was on the pundits’ prediction because those who bet assumed the media were the experts, or came to the same conclusion, or the starting odds were set on the basis of media predictions. I cleaned up – won over a thousand dollars – on 59, and the Poll Bludger and I were the only people writing anywhere whose predictions were in the range of 58-60 (I bet on that spread). Now, neither the Poll Bludger nor myself were “insiders” in any real sense – just close observers of politics and that particular election. So the markets reflected common wisdom in the media, and nothing else. I later found out that party polling would have formed the basis for a correct prediction, and thus a bet, but that knowledge (which would at the time have been restricted to maybe about 15 people) which would be the basis for an “insider” judgement was nowhere reflected in the betting markets. This seems to me to be a good measure of the thesis, because these outcomes are much finer than “win” or “lose”.

    (4) No doubt the media should give prominence to a range of indicators, and note the well known problems with polls. But I’m not certain of the justification for suggesting polls should be downplayed – if it’s a criticism of the fact that discussion of campaigns focuses on the “horse race” rather than the issues, I don’t see how giving more prominence to betting markets which also focus attention on the horse race in any way ameliorates things.

  2. If the betting markets are better than the polls but still not great because too many punters just bet based on the polls, does that mean we can improve on the markets by adjusting them away from the polls? Eg, if the polls say 60% chance of Lib win and the markets say 55%, should we conclude that the best guess probability is less than 55%, because the accuracy of (the smart money in) the markets have been diluted by the punters who are just betting on the basis of the polls?

  3. Andrew Leigh says:

    Mark/David – prices aren’t set by the average punter, but by the marginal punter. You can have plenty of dumb money in the market, and still get smart prices (same is true in horse betting, stockmarket, etc).

    if it’s a criticism of the fact that discussion of campaigns focuses on the “horse race” rather than the issues, I don’t see how giving more prominence to betting markets which also focus attention on the horse race in any way ameliorates things.

    Mark, IMHO, sportsplay journalism (who’s up, who’s down?) is fed by regular stories about changes in the polls. Since betting odds are more stable, they’re also more boring to reporters. If they could only report betting odds, they might turn to talking more about the issues.

  4. Geoff R says:

    This looks like one of those economists vs. non-economist battles. It all seems rather metaphysical. Neither polls nor markets deal with something that exists here and now but with what will happen in the future. But I am inclined to agree with Peter Brent that betting markets piggyback on opinion polls. When there is a dichotomy between markets and polls as until fairly recent we have punters making guesses as to how public opinion will change in the run-up to an election. Labor might be well ahead now but Howard is a better campaigner, voter will as the election approaches focus on pocket book issues etc., people like Beazley but when forced to make a choice between alternative PMs will prefer the cunning rodent etc. The more confident people are the more money they put on their bet. So I agree they don’t just follow polls. But how would markets perform without polls? We don’t have any historical examples. But one hint might come from close scrutiny of press commentary, if there had been a voting market back then it might have followed press commentary. My analysis of the 1930 and 1932 NSW elections found that press predictions, even when sourced to insiders, underestimated swings. The 1943 federal Labor landslide seems to have been unexpected. Insiders, and later punters, tend to be more conservative, compared to the outcome and to polls. Do markets add more than the rule of thumb to 1. follow the polls, 2. assume incumbent governments will do a bit better than the polls, 3. but if a clear shift of opinion has emerged in the polls over the campaign (Tasmania 2006, Qld 1995, Victoria 1999, Commonwealth 2004) consider that this will probably be held or even increase to election day? If Rudd wins where does leave all those who bet against Labor in 2006 defying the polls that had Labor ahead?

  5. Peter Brent says:

    Andrew, you quote me but address almost none of it. My position might be summed up as:

    (1) punters, like everyone else, get their “gut instinct” about what will happen at the next election mainly from the opinion polls. They don’t pull it out of their navels or from the pub. In the end, all these opinions come from the media, and much of that rests on opinion polls.

    (2) They’ve moved in Labor’s favour because the polls have been so pro-Labor. If we now get two opinion polls in a row showing the government ahead it’s London to a proverbial that the money will move back in Howard’s favour.

    As I said: “B-a-a-a-ah”

    (3) You say the pundits did better in 2004 than that panel of experts. But 70% of the experts got it right. You invest a wisdom in your pundits there is no evidence for, that they expected an increased majority.

    Correct me if I’m wrong, but there were no betting markets on margins. A reasonable guess would be that if there were, the money would have been on a decreased government majority. That’s what most people thought would happen.

    (I know you interpret the odds being greater in 2004 than in 2001 as evidence, but that doesn’t work for me.)

    Finally, no one really believes that a poll showing Labor ahead 58 to 42 literally means that is what an election result will look like in eight months. The pollsters don’t represent their numbers like that.

    So.

  6. Hans says:

    I haven’t read Andrew’s work on this either. But I have a question: are the betting odds that predict the result an average of the odds over the period running up to the election, or the odds on election night?

    If you’re watching a one-day cricket match, the odds fluctuate plenty during the game. The odds right near the end are a better predictor than the odds near the beginning, and better than experts’ early predictions.

    One’s view about the value of betting odds would be skewed to the positive if one took the odds on election eve?

  7. Andrew Leigh says:

    Peter, thanks for taking the time to respond. I had thought my parenthetical asides had addressed your post, but let me see if I can do a better job of persuading you by going through the critiques one by one.

    (1) punters, like everyone else, get their “gut instinct” about what will happen at the next election mainly from the opinion polls. They don’t pull it out of their navels or from the pub. In the end, all these opinions come from the media, and much of that rests on opinion polls.

    (2) They’ve moved in Labor’s favour because the polls have been so pro-Labor. If we now get two opinion polls in a row showing the government ahead it’s London to a proverbial that the money will move back in Howard’s favour. As I said: “B-a-a-a-ah”

    JW and I (and others) have done formal tests of this. Polls do affect betting markets a bit, but not much. And clearly the betting markets are not purely a reflection of the polls, or they would at some point have had Latham ahead in the last election cycle (he was always behind). Moreover, if this were true, then the betting markets would be as extraordinarily volatile as the markets, leaping from one party to another from month to month. Fact is, the betting markets are far more stable than the polls.

    (3) You say the pundits did better in 2004 than that panel of experts. But 70% of the experts got it right. You invest a wisdom in your pundits there is no evidence for, that they expected an increased majority. Correct me if I’m wrong, but there were no betting markets on margins. A reasonable guess would be that if there were, the money would have been on a decreased government majority. That’s what most people thought would happen. (I know you interpret the odds being greater in 2004 than in 2001 as evidence, but that doesn’t work for me.)

    You’re right – we have no marginals markets. I’m not sure why you think that the higher odds of the Coalition winning are not a useful indication that the markets thought there would be an increased majority. But if you think there’s no useful Australian evidence either way, let’s look to overseas evidence. In Expert Political Judgment, Philip Tetlock shows that US political experts do worse than ‘dart-throwing monkeys’. Meanwhile, in the US prediction markets, Intrade’s odds correctly forecast the outcome in all 50 states.

    Finally, no one really believes that a poll showing Labor ahead 58 to 42 literally means that is what an election result will look like in eight months. The pollsters don’t represent their numbers like that.

    I’d make two points on this. First, the pollsters represent their numbers as being accurate to within the margin of error – I think that’s unlikely. Do you really believe that the Coalition is ever a 1 in 10,000 chance to win an election? If not, then you should view any 56/44 poll as being afflicted by more than mere sampling error. Second, the pollsters represent their results as being a useful tool in forecasting the next election. You use them that way on your blog (as I do sometimes). All Justin and I do in our work is to put that contention to the test.

    On an unrelated note, I liked your comment in Crikey today.

  8. late night thoughts from California on this…

    I’m not persauded that we’ve thoroughly tested the relationship between polls and betting prices. Some thoughts: (1) there are multiple polls out there, each with sampling error and house-specific biases; (2) they measure/publish lots of things relevant to forming judgements as to who might win the next election (not just the aggregations of responses to the fanciful “if an election was help tomorrow…”); (3) polls give an estimate of something like the 2PP vote split, while the betting prices get translated into probabilities of a given party winning — the mapping between those two quantities isn’t that well defined, I think… Anyway, it might be nice to sort those things out empirically; for one thing, I tend to think we’d be able to make sense of why the betting markets didn’t reflect Labor’s lead in the polls in the early Latham days.

    Endogeneity: at some point, don’t the betting markets impact the polls, and indeed, doesn’t commentary on both also enter the process? One would imagine that the polls and the markets would have to be “cointegrated”, in that the two can’t be pointing in different directions forever (which from memory I think Andrew and Justin tested?). But taking a look at this, especially in light of the recent movements in both polls and markets might be interesting…

    But most of all, wouldn’t an experiment be wonderful (if impossible…): a election betting market without polls…? Surely these exist in the world, somewhere? They’d provide a strong test? I am sure one of the many papers on election markets must do this…no?

  9. Stephen Wood says:

    Just my two cents…

    Having read ‘The Wisdom of Crowds’, I think I understand the concept of the betting market being more successful in the long run than a panel of experts – the market contains more unique information, whereas the experts are likely to have very similar thinking styles and information. However, I still think that Andrew hasn’t addressed some of the points raised.
    First, over what time span are we talking for the accuracy of the betting market? In ‘The Wisdom of Crowds’, betting market success is judged on the night before the election (I think). Polls cannot possibly be taken that close, so the punters have the advantage of being sensitive to late swings.
    Second, polls today don’t predict the result of an election to be held in November (or whenever). They try to reflect the opinion of the voting public at the time of the poll. The margin of error is around the opinion levels, not the predicted winner.

    Plus, I think part of the problem here is ascribing a dichotomy to a probability in the betting markets.

  10. Rod says:

    Andrew (and Simon) – is it Germany or some other European country where polls are not allowed? That could be the place to test the theory. On this topic I think certainly that “margin” betting would consistently show people picking closer margins than actually occur (much like Football Grand Finals). Also I have often wondered what it must have been like in the lead-up to those elections like 1943 and 1946 when there were no polls (or were there?) – I know that if you only read all the media in the lead-up to the 1916 and 1917 plebiscites on Conscription you would have believed the “Yes” vote was heading for easy wins, and of course the “No” vote got up both times, with a bigger margin the second time.

  11. Chris says:

    (partial repost from another thread)

    Considering a normal distribution with sample size of 1000 (common for political polls), we’re looking at a margin of error of +/- 3% 19 times out of 20. Eliminating any questionnaire or sampling bias, that’s just cold statistical fact.

    The main problem with polls is that there is nothing at stake. Most research asks your opinion, and opinions have no consequence. Voting in an election is vastly different, as there is much at stake for the individual, whether ideology or a mortgage, not to mention plenty of noise from the campaign. A week is a long time in politics

    Thus, polls measure a mood accurately within statistical limits, but not necessarily the actual “booth vote” because they cannot adequately replicate an election voting scenario. But it’s close.

    Run the poll Friday night before an election and, within limits, you’ll be very close barring those who change their mind the night before or in the booth. These mind changers have never been measured, thus it’s an X factor.

    But in the end, if you woke up tomorrow and were told you had to vote that day, I can say with certainty that Rudd would scamper home. The mood is too strong at the moment. And at $1.80, I’d throw down some hard earned too.

  12. cam says:

    Maybe we should have betting markets on issues and policies then.

  13. Benno says:

    Andrew’s comment

    Do you really believe that the Coalition is ever a 1 in 10,000 chance to win an election?

    is not fair or accurate.

    To arrive at this figure he assumes that the poll is accurate and that 58% of people really will vote for the ALP. The chance of the Coalition winning a majority in a majority of seats with 42% of the TPP vote is then 1 in 10,000.

    Consider this somewhat irrelevant asside: in order to form government in a single-member chamber, a political party needs 50% of the vote + 1 vote in 50% of the seats. This means that it is possible for a party to win with barely more than 25% of the total TPP vote, (50% of 50%), or in the Australian HoR at the moment they would need 25.33% + 77 votes. obviously The probability of this happening is considerably less than 1 in 10,000.

    A better approach for Andrew is to first factor in the enourmous margin of error 2.6-3% 95%, or something like that for a 1200-2000 sample size, before he works out the probability of holding government with many marginal marginals while the other party has their votes tied up in huge majorities.

    Andrew also says that the polls swing madly while betting odds stay steady. I think Peter’s point is that punters have responded after seing a month of very favourable polls in Labor’s favour, not just a couple.

  14. Peter Brent says:

    Andrew,

    Thanks for reply.

    Agree to disagree.

    The polls are influencing one punter’s behaviour at least (the PM), leading to unforced errors, leading to more bad polls. A different story.

    Agree with Benno’s last par above.

    (Crikey comment looked harsher when I saw it in print.)

  15. Andrew Leigh says:

    Thanks for the many provocative comments. A couple of quick responses.

    1. Chris, I’m aware of how one can calculate the statistical margin of error. But that assumes that the poll is a random sample of the population. The point we make in our Economic Record paper is that the probabilities you infer from this (99.9% chance to win one week, 1% chance a few weeks later) are so unbelievable as to suggest that polls are not really a random sample, and therefore the true margin of error must be bigger. One explanation is low response rates. My guess is that pollsters’ response rates are <50%. For obvious reasons, no pollster reports them.

    2. Simon, the best test of betting markets that are unaffected by polls is a paper by Rhode & Strumpf. Before modern polling, the New York bookies ran betting markets. They pull the odds out of historical newspapers, and show that those markets were extremely accurate in predicting results of the 1884-1940 elections.

    3. Peter, I’m told that when JWH is following US elections, he asks for betting market updates. Lastly, I was thinking more about your Crikey comment on the way into work today. What was so good about it was its Obama-esque appeal to the best in our nature. There’s not enough of that in modern politics.

  16. Sacha says:

    If betting markets are better predictors of election results than opinion polls, well great. The mechanisms for this may be unclear but this opaqueness don’t mean that better markets aren’t better predictors.

    People mentioned predicting the actual number of seats won in an election (eg in Qld). Just a thought – there are often large numbers of seats, and local factors often play a big part (eg Bundaberg in the recent Qld election), and so without detailed local knowledge, predictions of the combined results in a lot of local seats may be less accurate than a feel for the overall result, although of course the overall result is an aggregation of a lot of local results.

  17. Peter Tucker says:

    My two bobs worth.

    Both polls and betting markets have their failings and their strengths. I am equally sceptical of both.

    I have read Andrew and Justin’s papers and agree with their logic that as a prediction tool betting markets might do a better job than polls. But the bookies still get it wrong a lot of the time. For a start, the betting markets don’t always follow the polls. If they did, Labor federally would have been the favourites through most of 2006. But they weren’t: despite Labor having a lead on a 2pp most of the year, Howard’s crew were clear, almost hot, favourites. It is only recently that the punters have woken up.

    The cause of Howard’s favoritism in the betting market throughout 2006 was the punters’ (and the bookies’) belief in the received wisdom (fed by many commentators and media analysts) that mid-term polling didn’t matter because voters would return to Howard once an election loomed. I have some sympathy for the “incumbency is king” as well as the “Howard is a great campaigner” theories too, but one still has to believe one’s eyes too.

    I think the polls during 2006 DO tell a story, particularly when looked back on. I write about that here: http://www.onlineopinion.com.au/view.asp?article=5494

    Great discussion.

    Sincerely

    Peter T

  18. Stephen L says:

    I think there is a difference between betting markets ability to predict who will win, and ability to predict margins.

    I haven’t seen many opportunities to bet on margins (particularly 2pp rather than numbers of seats won) but I think they would do very badly – people would usually understimate swings.

    I am also pretty sure there are biases in the betting markets – most people prefer to bet on who they want to win, and since Coalition supporters are richer than Labor supporters that may create a natural weight of money. Andrew’s papers may show that this bias is less than the various problems with the polls, but I’m pretty sure it is there, and systematic, where problems with polls may vary with time, in some elections favouring one party in some elections favouring another.

  19. Stephen L says:

    Also, no sensible person relies on individual polls – as Andrew notes they’re very volatile. A fairer comparison would be between a smoothed avarege of the polls over a reasonable period of time and the betting markets, rather than just taking the polls at a particular moment.

  20. derrida derider says:

    Of course, if pundits start adjusting their views to fit the betting markets, and betting markets in turn take account of pundit’s views, then the recursion is going to lead to stability problems. The predictive ability of the markets will break down.

    It’s another manifestation of Goodhart’s Law.

  21. Pingback: Club Troppo » Friday’s Missing Link

  22. John Quiggin says:

    I think it would be helpful to ask what you are claiming, and to put this in terms of the efficient markets hypothesis.

    Weak form efficiency (that markets do better than a prediction based solely on past election outcomes) is not too problematic

    The a priori arguments for semi-strong efficiency – markets do as well as possible, given available public information – are pretty convincing, and my feeling is that in financial markets, this is true most of the time, though there are occasional bubbles. Since (past) polls are public, you would expect markets to outperform a naive forecast based on a single past poll, which is the criterion you mostly seem to test.

    Finally, there’s strong form efficiency, that markets make optimal use even of private information. I think the evidence runs against this – markets don’t seem to give any warming of surprise outcomes, like the recent Austrian election where neither polls, pundits nor markets predicted the Soc*alist win.

  23. Pingback: Andrew Leigh » Blog Archive » The poll’s a longshot

Comments are closed.