Beware the Stunning Pilot Program

Kevin Drum looks at the results of a Chicago pre-school project and argues that this makes a slam-dunk case for taking this nationwide:

The results were especially good among children born to mothers who never finished high school: high school completion rates were roughly ten percentage points higher and rates of substance abuse and felony charges were roughly ten percentage points lower. Overall, the preschool groups had higher high school graduation rates, higher on-time graduation rates, higher college attendance, higher economic status, and higher incomes compared to the group who didn't attend preschool. Interestingly, the positive effects were limited to boys. Girls, however, responded more positively to school-age interventions.

So how much does this kind of preschool intervention cost? Let's do some rough back-of-the-envelope figuring. The cost per student was about $10,000, and nationwide there are about 4 million kids each at ages 3 and 4. So if you implemented this for the entire country for two years of intervention, it would cost about $80 billion. More realistically, if you limited it to, say, the third of the population most at risk, it would cost something like $25 billion. And if you count just the amount over and above what we already spend on existing preschool programs, it's more on the order of $15 billion.

Is that worth it? Well, out of 4 million kids, 2 million are boys and about 250,000 are children of mothers who didn't complete high school. Within this group, about 30,000 more would complete high school and 30,000 fewer would commit serious crimes and become drug abusers. That's per year. Fast forward 20 years from preschool and that adds up to about 300,000 kids between the ages of 16-25, the prime problem years. Just on the grounds of reduced crime and substance abuse within that group alone, this is money well spent. Add in all the other benefits, and doing something like this on a nationwide scale is a no-brainer.

If we could replicate the results of Chicago's Child-Parent Center (or Perry Pre-School, or Abecedarian), then yes, it would be a no-brainer.   But that's a pretty big hurdle.  As I've written before, it's a huge mistake to assume that a pilot program can be rolled out on a large scale.  Pilot programs are run by top-notch experts who are committed to, and familiar with, the organization's goalst.  Attentive to producing good data, they follow procedures much more closely than ordinary employees.  They have all the enthusiasm of someone embarked on a noble project--one of temporary duration.  They have none of the frustration and demotivation of people stuck in challenging but often tedious jobs.

With pilot programs, you always have to be on the lookout for the Hawthorne effect: people being studied often change their behavior in response to the fact of being studied, not to any particular intervention.  The effect gets its name from a factory where researchers were studying the effect of lighting on worker productivity.  What they found was that both raising and lowering the light level caused productivity to increase--the workers were responding to the researchers, not the lights.  It's not hard to imagine that a parent who is informed that their child is part of a Very Important Childcare Study might change their parenting in response.

When programs are rolled out on large scale, they suffer from a number of problems.  They cannot all be staffed with top-notch, motivated staff--Perry Pre-School, for example, required certified public school teachers with at least a bachelor's, while Head Start requires an associates degree.  The permanent staff will not hew as rigorously to procedures and methods as people doing a temporary experiment.  And of course, there is no Hawthorne effect.

What you want to look at is not pilot programs, but the programs with large-scale rollout.  And so far, research seems to show that the modest-but-still-exciting results we see from early childhood pilot programs are not replicated at the state or national level:

Reduced Grade Retention-Consistently, model programs and Head Start programs have been associated with a moderate to significant drop in grade retention. Research programs such as the High/Scope Perry Preschool Program and the Abecedarian Early Childhood Intervention report decreases in the number of children retained (held back) a grade in school (Schweinhart et al., 1993; Campbell & Ramey, 1995; Reynolds, 1996). Such models also report increases in the number of children who go on to graduate from high school, as compared to non-participating children. However, these are results seen in high-quality, intensive environments serving at-risk populations. These results have not been found systematically across all Pre-K programs. A meta-analysis of state-funded Pre-K programs found modest support for positive impacts in improving children's developmental competence in a variety of domains, improving later school attendance and performance, and reducing subsequent grade retention (La Paro & Pianta, 2000). 

Special Education Placement: Model programs such as the High/Scope Perry Preschool Program and the Abecedarian Early Childhood Intervention Program report a decrease in the number of children enrolled in special education classes. However, state-funded Pre-K program evaluations indicate a lack of positive impact on special education (La Paro & Pianta; 2000).  

Delinquency: Model programs such as the Perry Preschool Program and Chicago's federally-funded Child-Parent Centers report substantially lower rates of later law breaking and arrests among at-risk program participants than among non-participating children. An evaluation of a state-funded Florida program found fewer behavior problems as late as fourth grade. However, more generally state Pre-K programs report a lack of positive impact in prevention of later delinquency (La Paro & Pianta, 2000).

About the only thing that large programs definitely achieve is better results on grade advancement.  That's not nothing, but it's not much, either, unless it leads to higher graduation rates and lower delinquency.  It's probably not worth $10,000 a year, especially if that's not inflation adjusted (as in my experience, early childhood education studies usually aren't); that would be $27,000 in today's dollars, which would track with the costs of Perry Pre-School.

Now, of course, ideally we'd make the state programs more like the pilots.  But this is very hard to do, not just because of the limitations of scale, but because politics are going to get involved.  I'd like more proof-of-concept than a study of 1400 kids before I call this a "no brainer".