New Study Proves 9/10 Researchers Can't Distinguish Correlation from Causation (Or Why We Need a Better Press Corps)

A new study out of Flinders University looked at the number of friends that elderly people had in 1992. When they re-contacted the subjects in 2002, they found that those who had more friends were less likely to have died.

So far so good. But the researchers then concluded that having more friends makes you less likely to die. I admit that this might be true. But it might also be the case that healthy people are better able to maintain their friendship networks. As anyone who’s taken a single stats class knows, correlation≠causation. So researchers who only show correlations shouldn’t make unsupportable claims about causation.

This kind of study is a hardy perennial in Australia. The last example I can remember was a Commonwealth Bank Foundation study that found a correlation between financial knowledge and income, and concluded that knowing more about money made you richer (never once stopping to wonder whether rich people might have a bigger incentive to learn more about money management). Again, it might really be the case that knowing more about money makes you richer, but if you don’t have the evidence for it, you should at least acknowledge the possibility of reverse causation.

These aren’t hard issues to comprehend, but it’s amazing that almost no journalists are willing to approach research with a modicum of scepticism, while many researchers are willing to pull the wool over the public’s eyes. As a result, better studies that do try hard to show causation (eg. through randomised trials, natural experiments or instrumental variables) struggle to get airplay.

This entry was posted in Australian issues. Bookmark the permalink.

9 Responses to New Study Proves 9/10 Researchers Can't Distinguish Correlation from Causation (Or Why We Need a Better Press Corps)

  1. Sinclair Davidson says:

    You’re right, of course. My bug-bear is that many people seem to think correlation never equals causation. So, there’s a bimodal distribution, those who know nothing and those who believe nothing.

  2. Sinclair of course is right that correlation can equal causation. It would be worth looking at the actual study to see what’s been done with the stats – but usually these findings are publicised in advance of the journal publication, and by the time the journal comes out, everyone’s moved on. It probably adds some weight to the open-sourceish arguments about prepublication on the web for other researchers to have a look, but unfortunately in the social sciences it seems that the journal publishers like to lock up the articles to the greatest degree possible.

    I’ve made similar complaints about the media before – an introductory stats course – or at least something on how to interpret research – should be part of the training!

  3. Andrew Norton says:

    Whether correlation = causation, in Australian social science whatever the question the answer is government spending. This, from The Age:

    “Ms Giles said the results, published yesterday in the Journal of Epidemiology and Community Health, should encourage governments to provide infrastructure to allow elderly people to socialise.”

    The poor old dears. They can’t even make friends without the government helping them.

  4. Andrew Leigh says:

    Sinc & Mark – the article isn’t online yet, but a similar study by the same authors (http://jah.sagepub.com/cgi/content/abstract/16/4/517) makes clear that they’ve done nothing to correct for the fact that people who have friends are systematically different from those who don’t. Regardless of how many observable traits you control for, folks with more friends are always going to be different from those with less friends on unobservable dimensions. And if these unobservables are correlated with health (as they most likely will be), then any claims about causation are bunkum.

  5. Andrew, I wasn’t defending the specific study as such. Thanks for the info.

  6. What’s Nagi disability by the way?

  7. Sinclair Davidson says:

    Like Mark, I’m happy to believe you. No doubt the study is poor. My point is that two correlation and causation mistakes can be made: the type you’ve identified (probably the more important mistake) and the type I’ve identified (probably less important).

  8. Sylvia Else says:

    When you send the media to the course, can you make sure they’re properly instructed on the meaning of the word “significant” when used by researchers?

    Sylvia.

  9. Andrew Leigh says:

    Mark, one site describes Nagi disability scores as “a measure of ability to perform four basic functions of upper and lower extremities”. So I guess just motor function.

    Sylvia, I agree – though some would argue that we economists spend too much time worrying about statistical significance, and not enough on economic significance (coefficient magnitude).

Comments are closed.