This permanence does not necessarily match real-world conditions. Kids cycle in and out of street gangs the way they cycle in and out of any other social group, and many young men age out of violent behavior. Regularly purging the gang database, perhaps on a one-year or two-year cycle, would allow some measure of computational forgiveness. However, few institutions are good at keeping the data in their databases up-to-date. (If you’ve ever been served an ad for a product you just bought, you’re familiar with this problem of information persistence and the clumsiness of predictive algorithms.) The police are no worse and no better than the rest of us. Criminologist Charles Katz found that despite a written department policy in one large Midwestern police gang unit, data was not regularly audited or purged. “The last time that the gang unit purged its files, however, was in 1993—approximately 4 years before this study was conducted,” he wrote. “One clerk who is responsible for data entry and dissemination estimated, ‘At a minimum, 400 to 500 gang members would be deleted off the gang list today if we went through the files.’ Accordingly, Junction City’s gang list of 2,086 gang members was inflated by approximately 20% to 25%.”
Does current technology offer better alternatives? One way to gauge the level of sophistication in software knowledge is by browsing Github, a popular code-sharing platform. A search for the term “criminal database” reveals six different free, open-source database applications that anyone can download and use. None of them contain an expiration date, or any regulations about purging, or any kind of guidance on ethical use.
When I talk about the ethical responsibility of software programming, I usually get a question like, “If Amazon can predict what book I want to buy next, hasn’t this problem already been solved?” The answer is inevitably no. When computer scientists were building the Internet in the late 1980s, there weren’t any widely adopted or established ethical guidelines because we were building these systems for the first time in human history. The Association for Computing Machinery, the central professional organization for computer science, does publish ethical guidelines. They are recommendations, not requirements; following the guidelines is left up to individual programmers.
Now that the Internet is thirty years old, the long-term consequences of information permanence are becoming clear. We also need to acknowledge that computer systems are not a panacea. “Your program really does stink, and the sooner you get used to the idea, the better,” writes Nathan S. Borenstein in Programming as if People Mattered. “The inadequacies of your software are simply a reflection of your frail, shortsighted, and limited human nature. Every program ever built is doomed to eventual obsolescence.” We need to put people before programs, and if programs don’t reflect our human values, we need to change the code. And if programmers can’t write code that is fair and just, we should consider relying on people instead of programs.
If American law enforcement is going to go deeper into the brave new world of data-driven policing, we need to create systems that have human values embedded in them. If our technological systems are entrapping innocent citizens or tampering with the presumption of innocence, should they be used?