Smart statistics have changed baseball, health care, and many other industries. But why haven't they changed crime and punishment?
In the movie Moneyball, Oakland A's general manager Billy Beane (played by Brad Pitt) learned that the key to building a successful baseball team on a shoestring budget was deceptively simple: When signing a hitter, the crucial statistic was his on-base percentage. This flew in the face of the conventional wisdom that only an experienced scout could adequately size-up a player and pluck successful contributors from the bargain bin. But crunching the numbers revealed that paying a premium for a big-name player with lots of home runs was an expensive, ineffective way to build a team. On-base percentage, by contrast, was powerful but unglamorous -- and it was therefore undervalued by other teams. As a result, it became the key to the cash-poor A's success.
The importance of what happens between arrest and sentencing has gone largely unnoticed
This insight -- that smart statistical analysis can win baseball games -- has transformed how the sport is played. Technology has driven equally important shifts in other fields, including ones that traditionally did far less data collection and analysis than Major League Baseball. Innovative health care providers are examining electronic medical records to identify the neediest patients and provide them with better preventative care. In Camden, New Jersey, for example, a pilot program that targeted a small number of super-utilizers reduced this group's monthly emergency room visits by 40 percent and their hospital bills by 56 percent. In public education, technology can identify patterns in school data, allowing administrators to figure out which students are at the greatest risk of dropping out and to direct resources their way. Hamilton County, Tennessee, for instance, used this technique to reduce its dropout rate by 25 percent.
One area in which the potential of data analysis is still not adequately realized, however, is criminal justice. This is somewhat surprising given the success of CompStat, a law enforcement management tool that uses data to figure out how police resources can be used to reduce crime and hold law enforcement officials accountable for results. CompStat is widely credited with contributing to New York City's dramatic reduction in serious crime over the past two decades. Yet data-driven decision-making has not expanded to the whole of the criminal justice system.