Dan Hamermesh has a paper out on replication in empirical economics. It begins:
Economists treat replication the way teenagers treat chastityâ€”as an ideal to be professed but not to be practiced.
He quotes from several replication controversies, including what he regards as the best opening line in response to a comment:
The best model for this admission is Feldsteinâ€™s (1982) â€œReply,â€ the first sentence of which was, â€œI am embarrassed by the programming error that Dean Leimer and Selig Lesnoy uncovered but grateful to them for the care with which they repeated my original study.â€
And he makes what he calls “A Modest Proposal” (the Swift reference perhaps recognising its chances of success):
One arrangement consistent with the incentives I have discussed and that would generate additional replications would be for journal editors to commission leading senior empirical researchers to undertake a replication study of a paper of their choice, one that had previously been published in the journal. If editors of each of the three leading general journals commissioned two replication studies per year, with publication guaranteed subject to refereeing (NOT by the author of the original study) to assure some minimum quality level, more replications would be undertaken. Original authors would be expected to write a short reply to the final version of the replication study.
My own experience of the replication process (once as an original author having my data analysed, and twice as someone doing the re-analysis) is that this kind of thing is sorely needed. It would also be helpful if more academics put their research data and code on their websites, thereby encouraging colleagues – and students – to tinker with the results.