Grog Blog Slogged

Julie Robotham’s Dissection blog links to an article in the Canadian Medical Association Journal, purporting to show that giving out small amounts of alcohol to homeless men reduces harm to them. This may be true, or it may be false. But whatever your current view is on this question, I have no information to suggest that you should change it. This is the kind of article that should be given a weight of precisely zero. The sample size is 17, and there is no control group… a reminder that the publication bar in public health is ankle-high, if it exists at all.

This entry was posted in Uncategorized. Bookmark the permalink.

2 Responses to Grog Blog Slogged

  1. I’ve not read the article and, as a computer scientist, I have no reason (professional or otherwise) to defend publication practices in medicine. However, it is worth understanding how medical scientists come to be aware of a new medical phenomenon (an illness or condition, an effect, a treatment) requiring explanation or wider application: typically it is through an individual doctor (much less often, it is more than one doctor) noticing something which current medical science does not explain, or speculating that a particular condition may respond well to a novel treatment. This is how the effects of thalidomide were first noticed (by William MacBride), how the carcinogenic effects of peanut toxins were first noticed, and how numerous conditions and treatments were first brought to public attention. (It is why so many medical conditions have names, since they are named for the first victim or the first discoverer.)

    Now, if a doctor or a medical reseacher believes that there is a previously-unknown condition or effect or treatment, what is this doctor or researcher to do? He/she could adopt your (implied) approach and not publish anything about the effect or treatment until a statistically-valid study has been undertaken. Similarly, his or her peer reviewers or editors could refuse to accept for publication anything based on small sample sizes. But where will the money come from for a large-scale study? Certainly not from medical funding agencies or large medical charities, as they want to see published evidence of the existence of a new phenomenon or treatment before allocating their scarce research funds to its study or treatment. For treatments, pharmaceutical firms may be more willing to pay for something novel, but again they are unlikely to allocate funds to treatments or conditions so novel that they have never been reported in the medical literature. Would you allocate research funds to study some phenomena which only one person seems to know about?

    Thus, it is good and proper that medical journals publish information about conditions or treatments based very small samples, in order that medical science can be alerted to new phenomena worthy of further study. Since medical researchers now all take compulsory statistics courses, and indeed a whole discipline of medical statistics exists, only the simple-minded should have difficulty distinguishing between the validity of preliminary studies based on small sample sizes and that of fully-validated, large-scale studies.

  2. Andrew Leigh says:

    Peter, I think it’s an interesting point (in econometrics, we talk about this as the problem of balancing Type I and Type II errors). My fear is that if we allow the line between anecdote and research finding to blur too much, then we’ll be left in a world in which it becomes very hard to make scientific progress.

Comments are closed.