Correlation Leads, Causation Buried

The editorial in today’s Crikey email contains the best advice to journalists reporting on social science that I’ve seen in a long time.

The headline last week promised exciting news: “Geniuses ‘less prone to hangovers’.” Could it be that the same brain chemistry responsible for intelligence also mitigates the effects of heavy drinking?

Alas, no, or at least not from this study. There was no scientific experiment, just a survey asking about people’s drinking habits. Although “a higher IQ score was associated with a lower prevalence of hangovers”, the researchers suggest that more intelligent people “respond better to advice not to binge drink.” Who would have thought it?

Such blindingly obvious results are reported as news because ethical concerns prevent researchers from doing proper research on human subjects – force-feeding people alcohol in the interests of science is clearly problematic.

So researchers just conduct surveys instead. But surveys are generally unable to distinguish causation from simple correlation, not that this stops the media from happily publishing the absurd “news”. There were two prime examples just last month. One claimed that listening to explicit music made people more likely to have sex; another said that watching wrestling leads teenagers to fight more with their dates. Neither correlation is surprising, but neither is evidence in itself for a causal link.

Then yesterday up pops another one: a Reason Foundation study reported on an Age blog, showing that those who drink in moderation earn higher incomes than those who don’t drink at all, particularly if they drink socially. The researchers hypothesise that this might be due to the positive social capital effects of networking over a few drinks. It might. But did it not occur to them that people who earn more can also afford to drink more, and to do so in bars rather than at home?

If you don’t ask tricky questions, it’s much easier to get the answers you’re looking for.

I’m told that the author of the editorial is Charles Richardson. To ensure good research gets more media coverage than bad research, perhaps the Australian Research Council should fund him to deliver it personally to the nation’s tabloid journalists.

This entry was posted in Media. Bookmark the permalink.

10 Responses to Correlation Leads, Causation Buried

  1. Yobbo says:

    Force feeding people alcohol? I have participated in a few marketing studies where the task was to drink alcohol. If you offer payment you aren’t really forcing them are you?

    Are medical research laws really so strict that you couldn’t do this?

  2. Andrew C says:

    Might this perhaps expalin the academic prevalence for the campus bars and large dinner events at conferences.

    Of course that would assume higher IQ’s amongst academics which some of my fellow students might disagree with…

  3. Sacha says:

    It’s amazing that the difference between correlation and causation is continually not properly reported!

  4. Russell Hamilton says:

    Would it make any difference? If the newspaper articles screamed that listening to XXXX music is associated with XXXX, rather than causes it, it would mean the same to the general newspaper scanner. Does anyone have such a quality paper that they read it that closely, or much care what it says?

  5. It’s worse than above. Most of the “surveys” reported on do not even stand up to scrutiny enough to suggest correlation let alone causation.

  6. Claire says:

    “Such blindingly obvious results are reported as news because ethical concerns prevent researchers from doing proper research on human subjects – force-feeding people alcohol in the interests of science is clearly problematic.”

    That’s rubbish – it just implies that the researchers aren’t being very creative about experiment design.

    Anyway, a study like that would get passed by your average human subjects review as long as the participants were made aware of the risks, given the opportunity to leave the study at any point, and not requied to drive home afterwards :)

  7. Andrew Leigh says:

    RH/FXH/Claire, even if you only care about the quality of reportage, I think widespread reporting of dubious research creates the wrong incentives. If journalists ask more critical questions, it’ll be a small push to researchers to do cleverer work.

  8. derrida derider says:

    Anyway, a study like that would get passed by your average human subjects review as long as the participants were made aware of the risks [and] given the opportunity to leave the study at any point

    Umm, Claire, ever heard of selection and attrition biases? No, if our experiment is to pure we need to take random people off the street and, teetotallers included, physically force the grog down their throats.

    Last time I got home tired and emotional I explained to my wife that I’d been an unwilling participant in such a study, but she didn’t believe me.

  9. Andrew Leigh says:

    One of the interesting ways of handling this is to look at the effect of changes in alcohol taxes. Grog gets more expensive in a state, people consume less of it, and we get to see the results on their health/behaviour/etc. Here’s the abstract from one such study:

    This article presents evidence that sexually transmitted disease (STD) rates are responsive to increases in alcohol taxes and in the drinking age. The presumed relationship is that a more restrictive alcohol policy reduces alcohol consumption, which in turn decreases risky sexual activity. Reduced-form regressions of STD rates on state alcohol taxes for the years 1981-95 (with controls for state and year) indicate that a $1 increase in the per-gallon liquor tax reduces gonorrhea rates by 2.1 percent, and a beer tax increase of $.20 per six-pack reduces gonorrhea rates by 8.9 percent, with similar though more pronounced effects on syphilis rates. Quasi-experimental analysis of alcohol policy changes supports these findings and offers evidence that increases in the drinking age reduce STD rates among youth. The estimated external cost of alcohol-attributable STDs exceeds $556 million annually, a factor that could be considered in determining optimal alcohol policy.

  10. Claire says:

    Derrida Derider, yes of course I have, I deal with it all time. There are plenty of topics though, where you just have to control as much as possible for factors like thus, simply because the “pure” (ha) experiment is unethical. That was my point.

Comments are closed.