The editorial in today’s Crikey email contains the best advice to journalists reporting on social science that I’ve seen in a long time.
The headline last week promised exciting news: “Geniuses ‘less prone to hangovers’.” Could it be that the same brain chemistry responsible for intelligence also mitigates the effects of heavy drinking?
Alas, no, or at least not from this study. There was no scientific experiment, just a survey asking about people’s drinking habits. Although “a higher IQ score was associated with a lower prevalence of hangovers”, the researchers suggest that more intelligent people “respond better to advice not to binge drink.” Who would have thought it?
Such blindingly obvious results are reported as news because ethical concerns prevent researchers from doing proper research on human subjects – force-feeding people alcohol in the interests of science is clearly problematic.
So researchers just conduct surveys instead. But surveys are generally unable to distinguish causation from simple correlation, not that this stops the media from happily publishing the absurd “news”. There were two prime examples just last month. One claimed that listening to explicit music made people more likely to have sex; another said that watching wrestling leads teenagers to fight more with their dates. Neither correlation is surprising, but neither is evidence in itself for a causal link.
Then yesterday up pops another one: a Reason Foundation study reported on an Age blog, showing that those who drink in moderation earn higher incomes than those who don’t drink at all, particularly if they drink socially. The researchers hypothesise that this might be due to the positive social capital effects of networking over a few drinks. It might. But did it not occur to them that people who earn more can also afford to drink more, and to do so in bars rather than at home?
If you don’t ask tricky questions, it’s much easier to get the answers you’re looking for.
I’m told that the author of the editorial is Charles Richardson. To ensure good research gets more media coverage than bad research, perhaps the Australian Research Council should fund him to deliver it personally to the nation’s tabloid journalists.
Andrew Leigh's Archived Blog 2004-2010 
Force feeding people alcohol? I have participated in a few marketing studies where the task was to drink alcohol. If you offer payment you aren’t really forcing them are you?
Are medical research laws really so strict that you couldn’t do this?
Might this perhaps expalin the academic prevalence for the campus bars and large dinner events at conferences.
Of course that would assume higher IQ’s amongst academics which some of my fellow students might disagree with…
It’s amazing that the difference between correlation and causation is continually not properly reported!
Would it make any difference? If the newspaper articles screamed that listening to XXXX music is associated with XXXX, rather than causes it, it would mean the same to the general newspaper scanner. Does anyone have such a quality paper that they read it that closely, or much care what it says?
It’s worse than above. Most of the “surveys” reported on do not even stand up to scrutiny enough to suggest correlation let alone causation.
“Such blindingly obvious results are reported as news because ethical concerns prevent researchers from doing proper research on human subjects – force-feeding people alcohol in the interests of science is clearly problematic.”
That’s rubbish – it just implies that the researchers aren’t being very creative about experiment design.
Anyway, a study like that would get passed by your average human subjects review as long as the participants were made aware of the risks, given the opportunity to leave the study at any point, and not requied to drive home afterwards 🙂
RH/FXH/Claire, even if you only care about the quality of reportage, I think widespread reporting of dubious research creates the wrong incentives. If journalists ask more critical questions, it’ll be a small push to researchers to do cleverer work.
Anyway, a study like that would get passed by your average human subjects review as long as the participants were made aware of the risks [and] given the opportunity to leave the study at any point
Umm, Claire, ever heard of selection and attrition biases? No, if our experiment is to pure we need to take random people off the street and, teetotallers included, physically force the grog down their throats.
Last time I got home tired and emotional I explained to my wife that I’d been an unwilling participant in such a study, but she didn’t believe me.
One of the interesting ways of handling this is to look at the effect of changes in alcohol taxes. Grog gets more expensive in a state, people consume less of it, and we get to see the results on their health/behaviour/etc. Here’s the abstract from one such study:
Derrida Derider, yes of course I have, I deal with it all time. There are plenty of topics though, where you just have to control as much as possible for factors like thus, simply because the “pure” (ha) experiment is unethical. That was my point.