What’s the Evidence on Evidence Based Policy?

Posted on April 1, 2010 by andrewleigh

Last year, the Productivity Commission ran an event on the topic ‘Strengthening Evidence-based Policy in the Australian Federation’, of which I was one of the participants (my contribution was titled: ‘Evidence-based policy: summon the randomistas?’). The PC has now produced a mighty two-volume set, which is also available on their website. If you’re short on time, go to the background report (volume two), which is likely to become a standard reference for anyone thinking about evidence-based policy in Australia. We’ll see if George Pell gets around to reading Appendix A.

This entry was posted in Australian Politics, Economics Generally, Randomisation. Bookmark the permalink.

4 Responses to What’s the Evidence on Evidence Based Policy?

  1. Don says:
    April 4, 2010 at 10:26 pm

    I was interested in your response to Patricia Rogers.

    It’s certainly true that “Australia is in ‘no danger of over-relying on randomised trials’, or blindly accepting them as the only method of evaluation.” But there’s still the issue of whether other kinds of evidence should count.

    Writing about the Moving to Opportunity experiments, sociologist Robert Sampson argued:

    “Many causal conclusions, including the consensus that smoking causes cancer, have come about after years of careful observational research linked to rigorous thinking about causal mechanisms. The early discovery of penicillin and the cause of cholera outbreaks were similarly observation based.”

    Click to access 2008_AJS_Moving_to_Inequality.pdf

    Sampson’s interest is in neighbourhood effects. And since the unit of analysis here is an entire community, it would be very bad news indeed if RCT were the only way to know whether an intervention worked.

    If we placed too much weight on experiments, we’d risk devoting too much effort to interventions that target individuals than those that target communities and social processes.

    I think it’s worth looking at whether alternatives to experiments can deliver useful results.

  2. Andrew Leigh says:
    April 6, 2010 at 11:20 am

    Don, your concern is an important one – randomised experiments give precise causal inference, but suffer from drawbacks relating to spillovers and generalisability. So I agree that we’d never want 100% of our evaluations to be randomised.

    In practice, with less than 1% of evaluations being randomised, there’s virtually no chance that we will place too much weight on experiments anytime in my lifetime.

  3. conrad says:
    April 6, 2010 at 1:17 pm

    “but suffer from drawbacks relating to spillovers and generalisabilit”
    .
    They’re also exceptionally expensive to run, so there’s a huge opportunity cost in running them (do you want a series of well controlled non-randomized experiments that are fast to run or a single randomized experiment that takes ages? I guess it depends both on what you are looking at and what funding agencies are willing to fund).

  4. econjeff says:
    April 12, 2010 at 10:59 am

    Seems to me that Chapter 4 of Volume 1 is the real keeper. 🙂

Comments are closed.