Where are the Replicants?

Dan Hamermesh has a paper out on replication in empirical economics. It begins:

Economists treat replication the way teenagers treat chastity—as an ideal to be professed but not to be practiced.

He quotes from several replication controversies, including what he regards as the best opening line in response to a comment:

The best model for this admission is Feldstein’s (1982) “Reply,” the first sentence of which was, “I am embarrassed by the programming error that Dean Leimer and Selig Lesnoy uncovered but grateful to them for the care with which they repeated my original study.”

And he makes what he calls “A Modest Proposal” (the Swift reference perhaps recognising its chances of success):

One arrangement consistent with the incentives I have discussed and that would generate additional replications would be for journal editors to commission leading senior empirical researchers to undertake a replication study of a paper of their choice, one that had previously been published in the journal. If editors of each of the three leading general journals commissioned two replication studies per year, with publication guaranteed subject to refereeing (NOT by the author of the original study) to assure some minimum quality level, more replications would be undertaken. Original authors would be expected to write a short reply to the final version of the replication study.

My own experience of the replication process (once as an original author having my data analysed, and twice as someone doing the re-analysis) is that this kind of thing is sorely needed. It would also be helpful if more academics put their research data and code on their websites, thereby encouraging colleagues – and students – to tinker with the results.

This entry was posted in Economics Generally. Bookmark the permalink.

8 Responses to Where are the Replicants?

  1. conrad says:

    Actually, in psychology and cognitive neuroscience, almost all (psychology) and some (CN) journals force you to at least make your data available for 5 years (and some of those data sets are expensive to collect), although if you ask people, people can just forget to reply to their email excluding one or two journals that have their own data repository and there isn’t much you can do about it. The obvious reason (excluding not wanting people to be able to do similar research as you) people often don’t give their code for complex simulation models away is that the code is often disgusting, runs of weird platforms etc. and giving it away would generate a large amount of queries and work.

  2. Paul Frijters says:

    you’re not thinking like an economist. What is the benefit of a journal having one of the papers it published exposed as wrong? Strongly negative in terms of reputation! What’s the cost? Substantial. So there’s no reason to encourage replication of the papers in ones’ journal. What about replicating stuff in other journals? Running replication wars with other journals (publishing falsifications of their papers) gets you into an open shit-fight and will cost you as editor in your attempt to get your own personal papers into the other top journals (since its not bon-mot to publish in your own journal). Who has the incentives to increase replication studies? No journal of importance. Just like a ministry has little incentive to evaluate the program it already universally implemented because it can only loose from the outcome, journals dont have incentives to increase doubt over their published articles.

    Just to illustrate why people dont want their mistakes exposed: may I remind you of your salacious claim a couple of blogs ago that baseball is multi-ethnic because it doesnt require a particular body shape? I notice you didnt want to comment on the matter further when pressed to replicate this logic for football (what ethnicty has the wrong body shape for football?), presumably because you realise you shouldnt have said that comment in the first place. If you were editing these pages and the comments on it ……. capiche?

  3. Andrew Leigh says:

    Paul, I think you’re missing the point – Dan is dealing precisely with the incentives. That’s why he’s making a proposal.

    As to comments, I’m sure you’ve noticed that I don’t jump in to respond to every critical comment. In many cases, I enjoy watching a conversation develop between commenters, rather than just having a 2-way dialogue.

    In the case of your particular comment, you took a statment I made (baseball is very ethnically diverse, maybe this is because body shape doesn’t matter), inferred something I didn’t say (the only factor affecting a sport’s ethnic diversity is body shape), and then jumped on this with apparent glee. Every now and then, it’s worth letting one go through to the keeper.

  4. Sinclair Davidson says:

    I’m surprised that people who claim to be scientists (or that economics is a science) would argue against replication. Afterall, isn’t that the ‘self-correcting’ argument we hear so much about. Paul is telling us economics has no incentive to self-correct. We should conclude then that (a) economics is not a science, and/or (b) economics is riddled with error.

    (b) is almost certainly true. I propose ‘externality’, ‘market failure’, ‘IS-LM-BP’, ‘S-C-P’, just off the top of my head as errors.

  5. Paul Frijters says:


    Neither Dan nor you address the incentive issue (i.e. there’s no incentive for the editors to go along with his plan) – a cardinal sin in the economics of desgning institutions.

    As to your the ethnicity business, let me replicate your pinstripe blog that said “Presumably this has something to do with the fact that you don’t have to have a certain body shape to be a good baseball player. ”

    it was a bad sentence to initially write and methink your wriggling above perfectly illustrates why economic journals are not interested in replication studies of their mistakes…

  6. Damien Eldridge says:

    Andrew, a version of Hamer’s paper that can be read by people without access to NBER publications is available on his website. The fo0llowing is a link to the paper:

    http://www.eco.utexas.edu/faculty/Hamermesh/Replication.pdf .

    Paul, why would replication reduce the reputation of a journal? If anything, I would think that replication would improve the reputation of a journal. After all, it is unlikely to result in authors being less careful in their empirical work. Furthermore, it is likely to deter explicit fraud (although I hope that is not a problem in economics).

    One issue that needs addressing in some fields of economics is the proprietory nature of the data. I suspect that this is a much bigger problem in emprical IO than it is in empirical labour economics, although it probably occurs in labour economics on occassions. While standard data sets such as the NLSY, the PSID, the CPS, HILDA, various ABS surveys and the like are readily available, some data sets are not. Are we going to refuse publication to studies based on proprietory data or should they be exempted from replication rules?

  7. Andrew Leigh says:

    Damien, thanks – I’ve updated the link. As to data access, most journals have rules about making code available, and promising to help replicators get access to proprietary data.

  8. christine says:

    No-one likes to make mistakes, including in accepting a bad paper for publication. I’ve come across a couple of papers recently that should never have been published due to howling errors that any decent MA (or Australian honours) student should have picked up on, even without actually replicating the study. What editor really wants to admit to such things happening on their watch, esp if they know no-one else is ‘fessing up?

    It seems to me that the incentives against bothering to do replications are a bigger issue. Hamermesh’s proposal would help deal with those incentives at least. Some commitment to the idea of replication as valuable would be nice at least. I was recently hit by a case where a journal refused to consider a submission precisely because it was a comment (read: replication ) on an article previously published in the same journal. No other reason given, but I must say my opinion of the journal went down a few notches after that.

    Besides the proprietory data, there are problems too with a whole lot of supposedly-accessible data. Anyone from outside Canada is going to face formidable barriers to replicating studies using Statistics Canada data – some physical (you must be located in a special facility to get the data), some monetary. If I’ve paid $5000 to get data, and you could pay $5000 to get it too but don’t want to, should I nonetheless be required to give it to you? You’re almost certainly not going to get a grant to buy the data to do a pure replication. And if I’m required to give you the data, how does this fit in with the big copyright signs plastered on the data I purchased? These are not insurmountable obstacles, but they do need to be considered nonetheless.

    And why has no-one commented on Hamermesh’s final line? Has he actually himself studied the effect of abstinence-only programs on teenagers’ sexual behaviour? If not, why the gratuitous comment? (PS: can’t tell whether he has in fact or not, since honestly his CV is too long for me to check completely, esp given this is a joke)

Comments are closed.