Schools That Transform

David Brooks reports on some surprisingly large impacts from a randomised trial of New York “Harlem Children’s Zone” schools. The effects are so big that I find it hard to imagine they’re readily replicable, but it’s tantalizing stuff.

The fight against poverty produces great programs but disappointing results. You go visit an inner-city school, job-training program or community youth center and you meet incredible people doing wonderful things. Then you look at the results from the serious evaluations and you find that these inspiring places are only producing incremental gains.

That’s why I was startled when I received an e-mail message from Roland Fryer, a meticulous Harvard economist. It included this sentence: “The attached study has changed my life as a scientist.”

Fryer and his colleague Will Dobbie have just finished a rigorous assessment of the charter schools operated by the Harlem Children’s Zone. They compared students in these schools to students in New York City as a whole and to comparable students who entered the lottery to get into the Harlem Children’s Zone schools, but weren’t selected.

They found that the Harlem Children’s Zone schools produced “enormous” gains. The typical student entered the charter middle school, Promise Academy, in sixth grade and scored in the 39th percentile among New York City students in math. By the eighth grade, the typical student in the school was in the 74th percentile. The typical student entered the school scoring in the 39th percentile in English Language Arts (verbal ability). By eighth grade, the typical student was in the 53rd percentile.

Forgive some academic jargon, but the most common education reform ideas — reducing class size, raising teacher pay, enrolling kids in Head Start — produce gains of about 0.1 or 0.2 or 0.3 standard deviations. If you study policy, those are the sorts of improvements you live with every day. Promise Academy produced gains of 1.3 and 1.4 standard deviations. That’s off the charts. In math, Promise Academy eliminated the achievement gap between its black students and the city average for white students.

Let me repeat that. It eliminated the black-white achievement gap. “The results changed my life as a researcher because I am no longer interested in marginal changes,” Fryer wrote in a subsequent e-mail. What Geoffrey Canada, Harlem Children’s Zone’s founder and president, has done is “the equivalent of curing cancer for these kids. …

These results are powerful evidence in a long-running debate. Some experts, mostly surrounding the education establishment, argue that schools alone can’t produce big changes. The problems are in society, and you have to work on broader issues like economic inequality. Reformers, on the other hand, have argued that school-based approaches can produce big results. The Harlem Children’s Zone results suggest the reformers are right. The Promise Academy does provide health and psychological services, but it helps kids who aren’t even involved in the other programs the organization offers.

… Basically, the no excuses schools pay meticulous attention to behavior and attitudes. They teach students how to look at the person who is talking, how to shake hands. These schools are academically rigorous and college-focused. Promise Academy students who are performing below grade level spent twice as much time in school as other students in New York City. Students who are performing at grade level spend 50 percent more time in school.

For more wonkish readers, the abstract of the paper is over the fold.

Are High-Quality Schools Enough to Close the Achievement Gap? Evidence from a Bold Social Experiment in Harlem
Will Dobbie and Roland G. Fryer, Jr.
Harlem Children’s Zone® (HCZ) is arguably the most ambitious social experiment to alleviate poverty of our time. We provide the first empirical test of the causal impact of HCZ on educational outcomes, with an eye toward informing the long-standing debate whether schools alone can eliminate the achievement gap or whether the issues that poor children bring to school are too much for educators to overcome. We implement two identification strategies. First, we exploit the fact that HCZ charter schools are required to select students by lottery when the demand for slots exceeds supply. Second, we use the interaction between a student’s home address and cohort year as an instrumental variable. Both approaches lead us to the same story: Harlem Children’s Zone is enormously effective at increasing the achievement of the poorest minority children. Taken at face value, the effects in middle school are enough to reverse the black-white achievement gap in mathematics and reduce it in English Language Arts. The effects in elementary school close the racial achievement gap in both subjects. Harlem Gems and The Baby College®, the only two community programs in HCZ that keep detailed administrative data, show mixed success. We conclude by presenting three pieces of evidence that high-quality schools or high-quality schools coupled with community investments generate the achievement gains. Community investments alone cannot explain the results.

This entry was posted in Economics of Education, Inequality. Bookmark the permalink.

9 Responses to Schools That Transform

  1. Invig says:

    Not readily replicable?!?

    Shame on you Andrew! Where is the willingness to build theory to explain these results? To uncover deeper truths and hard-to-measure variables?

    Nope, none of it. Only a dismissal of a ‘statistical blip’ that you cannot deploy your econometric toolbox on.

    I name you fraud.

  2. Pat K says:

    Steve sailor argues that’s a sloppy conclusion Brooks reached. Sailor appears to be right while Brooks and the Harvard professor seem to be incorrectly reaching such a conclusion through faulty analysis

  3. Invig says:

    I don’t know why there is a concern for disparity between blacks and whites – so long as individuals are able to reach their potential. If it so happens that black people (as a race) are less capable in maths, then that’s the reality. So what? I don’t judge the value of people on their ability to do maths.

    So you don’t need a white ‘control group’ – the control is supplied by the historical results the school had achieved thus far. If whites were improved by the same amount, and the gap was sustained; that’s fine. Its the overall improvement that’s important.

    People are different. Women are different to men, nerds are different to jocks, perhaps whites are different to blacks in some intrinsic way? But, you know, I doubt it. Why? Look at footage of black activists in the 60’s. They are some of the most articulate and impressive people I have ever seen speak. This includes Australian Aboriginals in Redfern over that period. Look at them now…

    Have we not systematically de-selected for brilliance and bravery in black society? In the worst case by killing the best (the black panthers) but otherwise not giving them the opportunity nor environment to flourish. By breaking the hearts of their parents. By grinding them down.

    That’s the stuff that affects the maths scores of today I would posit, but I would challenge an economist to include them in his study.

  4. conrad says:

    “Steve sailor argues that’s a sloppy conclusion Brooks reached”
    Why not ask Alan Jones what he thinks too?
    I agree with Invig, if you get good results with one cultural group, the correct comparison is the same cultural group in a different situation, not a group of middle class white kids who already go to decent schools and live in decent neighborhoods.

  5. Pat K says:

    the thing is that racial scores are closely monitored in the US across racial groups and even if you don’t think it’s important the apparatus of the education establishment thinks the so called divide is closed.

    Conrad, Steve sailor is generally good at figuring out the nonsense based stats.

  6. Invig says:

    But surely there is a distinction between what the ‘education establishment’ wants to achieve and what is a significant experimental result.

    To confuse the two seems incredible to me.

    One is the product of politics and ‘groupthink’ etc. etc. and, while not undesirable, does not mean that the underlying mechanisms allowing these black children to improve their test scores are somehow chance or non-reproducible.

    Pat K, I wish you had a blog so I could go all the way through it and crush other arguments you and your fellow travellers might care to mount. Based upon this instance, it would appear to represent thinking that actively opposes positive, systemic change.

    I mean, Andrew up there seeks the perfect double-blind (or whatever) trial (which he will never get) but at least he is open to the ‘tantalizing possibility’ that this experiment raises. You and your friend Steve go a step further and strike it down on spurious grounds. Because it does not comply with some random bureaucratic imperative?!?

    Do you have any idea what would happen to the scientist (who actually can mount the ideal trial) who rejected his experimental result because it didn’t comply with the government’s desires? He would be laughed out of his faculty as a complete fraud!!!

  7. Pat K says:

    I’m not a fellow or a traveler 🙂 and to be honest you seem to be setting up a strawman.

    Whether you like it or not the US educational establishment is primarily focused on closing the racial gap. It has nothing to do with what you and I would prefer. It’s simply a fact of life.

    The study doesn’t really appear to be closing what the establishment wants to close and what Brooks is wishing for.

  8. Invig says:

    Yeah I did get a bit full on there…

  9. conrad says:

    “Conrad, Steve sailor is generally good at figuring out the nonsense based stats”
    Generally with nonsense of his own. For example of the points he makes, some appear factually correct. For example:
    1) Black scores have been going up faster than white scores. You can see this even in IQ scores. (Look up Flynn effect and wikipedia)
    2) It isn’t simple to get gains of .3-.4 SDs in classes. Who said it was?
    3) Middle class white kids already have more spent on their education than poor black kids. How does spending more on white kids work as a control? How about controlling for environmental conditions etc. too?
    4) If you really care about the difference, then a slightly more intricate look at the SAT scores for mathematics (which have actually been going down in the last few years, which is conveniently ignored) will give a much more complicated picture. What you will find is that white male scores are going down but female ones are going up. So we would expect black males and white males to get more similar and black females and white females to diverge.

Comments are closed.