For Better Policymaking, Toss a Coin

I have an oped in the Australian Financial Review today, discussing the upsurge of randomised trials in development economics, and linking it to Indigenous policy in Australia. It was written under somewhat unusual circumstances. I’ve spent the past few days on a family holiday in Lamington National Park, near the NSW/Qld border. I had thought that it would be possible to find a phoneline, or get my mobile to connect to my laptop, but eventually I jumped in the car and drove half an hour through the forest to find an internet connection. Anyhow, the oped is over the fold.

First, Find out What Works, Australian Financial Review, 4 October 2007

Among the most interesting debates in economics today is the dispute between Jeffrey Sachs and William Easterly over how best to help the world’s poor. The discussion is interesting not only because it concerns the most important question in all of economics, but also because Sachs and Easterly happen to be exceptionally good communicators.  And like the longtime Mets-Yankees rivalry, both happen to be at New York universities: Sachs at Columbia University, and Easterly at New York University. 

At stake is the question of whether development economics requires a Big Push or piecemeal reform. In The End of Poverty: Economic Possibilities for Our Time, Sachs argues that the world’s least developed countries are caught in a poverty trap, and that success requires a doubling of foreign aid. In The White Man’s Burden: Why the West’s Efforts to Aid the Rest Have Done So Much Ill and So Little Good, Easterly contends that much foreign aid has been wasted, and that it would be better to focus on taking modest, deliverable steps to make poor people’s lives better.

Whoever is right (my own view is that the truth lies somewhere between the two), the Sachs-Easterly debate has helped to galvanise a revolution in development economics. Integral to the development economics revolution has been the advent of rigorous randomised trials to find out what works. Long regarded as the ‘gold standard’ in medicine, a spate of randomised trials are taking place across the developing world.

In Kenya, researchers have shown that deworming drugs dramatically reduce school absences, while worm-prevention education programs had no impact whatsoever. In the Philippines, savings incentives have been shown to boost the assets of the poor. And in India, a program that paid teacher bonuses raised student test scores by a significant margin.

The reason randomised trials are so powerful is that they allow us to know the counterfactual: what would have happened if the program was not implemented? Because participants are assigned to the treatment and control groups by the toss of a coin, we can be sure that any systematic differences in outcomes are due to the program itself.

Beyond program evaluation, randomised trials have also been used to help researchers learn more about issues of governance, which have long bedevilled development efforts. In a famous randomised trial in Benin, researcher Leonard Wantchekon persuaded presidential candidates in the 2001 election to randomly alter their campaign strategies: giving high-minded public policy speeches in some villages, and pork-barrelling clientelist speeches in other villages. He found that candidates won more votes when they delivered a clientelist speech, and that male voters tended to be more easily swayed by offers of local patronage than female voters.

More controversially, Marianne Bertrand and co-authors recently followed a group of Indian applicants through the notoriously corrupt process of obtaining a driving licence. One group were offered a financial reward if they obtained their licence fast, while others were not. The researchers found that the payment did indeed lead applicants to get their licences faster, but their skills were no better. Given a surprise driving test afterwards, 69 percent failed. Evidence from the randomised trial helps understand how corruption in the licensing process operates, and may assist reformers in improving the system.

As the results from these and dozens of other randomised trials begin to pour in, development economists will continue to hone their knowledge about what works best, and whether the kind of grand strategy of Sachs or the more piecemeal approach of Easterly is likely to offer more promise for raising the living standards of those in developing countries.

Yet while randomised policy trials are transforming development economics, it is startling that our homegrown development challenge – raising the living standards of Indigenous Australians – has seen so little attention to rigorous evaluation. With Indigenous life expectancy at third world levels, too much of Indigenous policy seems to be driven by instinct, and too little by results.

Instead of digging in behind their favoured solutions, wouldn’t it be better if both sides of politics admitted that most social policies aimed at improving Indigenous living standards have failed? Once we acknowledge how little we know, we might then begin by applying the same gold standard evaluation technique to Indigenous social policies that we apply to new drugs.

A dozen randomised trials of Indigenous social programs would cost comparatively little, but provide valuable new evidence. In health and education, employment and family policies, Indigenous policymaking could do with a little less rhetoric and a little more of the modest searching that characterises the best of development economics today. Think your policy works? Let’s toss a coin and find out.

Dr Andrew Leigh is an economist in the Research School of Social Sciences at the Australian National University.

This entry was posted in Indigenous Policy, Trade & Development. Bookmark the permalink.

8 Responses to For Better Policymaking, Toss a Coin

  1. Kevin Cox says:

    Andrew you will not be surprised to hear me suggest that creating new currencies is a good way to help implement trials. We believe that creating new currencies will help implement a particular policy be it reducing greenhouse gases or removing water restrictions. Currencies are an efficient way to implement policy rules and regulations.

    You can run randomised trials but you can also run real world alternative trials on different communities by creating a new currency and having different currency rules for either whole communities or for random samples within the community.

  2. Simon says:

    I think this is great, and something that should have been said – and done – a long time ago in Australia.

    However, even before we get to randomised interventions, I think there would be exceptional value simply in conducting a baseline survey of indigenous communities. My understanding, albeit shallow, is that the data available is limited, and not generally amenable to panel analysis (eg the census data). Even before we get to randomisation, I would be interested just to be able to see some basic summary statistics and to control for fixed effects; I think we could learn a lot about indigenous opportunity and returns to investment in indigenous communities by doing that, even if the identification is not as clean as a best-case randomisation would allow.

    A wide-scale baseline survey would be relatively cheap and would allow substantial efficiency gains in the design of future policy for indigenous standards of living; but a cynic might wonder whether that’s what indigenous policy has ever really been about.

  3. Vic says:

    AL wrote
    “Instead of digging in behind their favoured solutions, wouldn’t it be better if both sides of politics admitted that most social policies aimed at improving Indigenous living standards have failed?”

    Totally agree. But admitting this in itself would not create the kind of productive dialogue and policy framework required.

    Many of the solutions were not that bad, just badly articulated.
    However I also believe that these policies were developed within a framework of liberal democracy and notions of equity with wider Australians instead of a process of reparation and restorative social, economic and political justice.

    Until this happens, policies will always fail simply because they are so intimately controlled by a political culture that was itself formed from a dispossession of Indigenous peoples.

    For me this crisis is not about Indigenous people (althought they suffer the most from this crisis) but rather a crisis in how the most important institutions in this nation have yet address their own fidiciary duties to the first peoples of this nation.

    The result in the administration of Aboriginal affairs and the associated politics is that critiques from Left and Right have consistently been restricted to the management paradigm. This focus on administrative content and process to the exclusion of structures and values has created false oppositions in politicking about Aboriginal policy.

  4. Andrew Leigh says:

    Simon, the 1994 & 2002 NATSISS has significantly improved data availability in this regard – though you’re right that panel data would help us a lot (for 2 reasons, I think: Indigenous people move house more than non-Indigenous people, and people can self-define in to being Indigenous).

  5. Adam McCarty says:

    I am Chief Economist at a Vietnamese company that has done some “gold standard” – and many more below – baseline surveys and evaluations in Vietnam, Laos, Cambodia and East Timor. The general problem is that efforts to close the “evaluation gap” (see must face the on-ground reality of input-focused bureaucrats pushing through projects, and with no self-interest in revealing failures. The policy experiments trend highlighted by Andrew is led by a small group of World Bank and American academics. It is unlikely to feed into the reality of how aid is designed and delivered because it runs counter to the incentive structures of the givers and receivers – who also do not understand the importance of trying to answer the counterfactual question. Genuine ignorance combined with self-interest will ensure that the “gold standard” approach remains a fad unless something extraordinary happens at the political level.

  6. Pingback: Benin » Blog Archives » Here we go again!

  7. Vic says:


    Yes, policy development never follows the orthodox pathways between problem definition and implementation – but its often defended using this framework.

    One need only critical examine the Productivity Commission’s work/ assessment of Indigenous funding to see how entrenched self interest and ignorance is within policy formulation.

    It is interesting to note that while most of discussion on Indigenous policy is focused on past failure ie, failure as the construct – rather than as a starting point.
    The framework will begin its cycle again and the problem definition will not be focused on the framework and cycle itself ie, problem identification, policy analysis (solutions) but rather on variations of the same.

    What’s that saying about insanity – — doing something the same way over and over and expecting a different outcome?

  8. Pingback: Randomised trials and development policy « John Rawnsley

Comments are closed.