Colvin’s case highlighted another problem with MiDAS: It was designed with insufficient attention to the right to due process. It’s a common problem in automated systems, said Jason Schultz, a law professor at NYU and a legal and policy researcher for the school’s AI Now Institute. “Even though the systems claim to notify people, they typically only notify them of the end result,” he added. “They don’t usually give them any information about how to actually understand what happened, why a decision was made, what the evidence is against them, what the rationale was, what the criteria were, and how to fix things if they’re wrong.”
Although MiDAS had several steps for notification, the forms were confusing, leading some innocent people to self-incriminate, according to Steve Gray—then the director of the University of Michigan’s clinic—who described the problem in a co-authored letter to the U.S. Department of Labor. Moreover, many people never saw the messages. The first one, the questionnaire, appeared in an online account that most people checked only when they were receiving benefits. And by the time the second notification, by letter, came around, many people had moved to a new address, according to a 2016 report from Michigan’s auditor general.
Brian Russell, an electrician from Zealand, Michigan, learned about his alleged fraud only after the state took $11,000 from his 2015 tax refund. He visited a state unemployment-agency office and was astounded to learn that the government would take even more: “They were charging me $22,000, and I had no idea what was going on.”
Russell said he tried everything he could think of to figure out why he was being charged—he visited the nearest unemployment office and called the agency “probably hundreds of times,” but the phone system was overloaded. He’d wait for hours on hold. (The state auditor’s report found that of the more than 260,000 calls made to the line during business hours in a one-month period in 2014, about 90 percent went unanswered.) Once Russell finally got a person on the line, he said he was told he’d missed the 30-day window to appeal.
The University of Michigan’s law clinic eventually appealed Russell’s case to an administrative-law judge, who dismissed Russell’s charge. The agency refunded most of the money. (Russell said he is still owed about $1,500.) But in the meantime, he said, he had to declare bankruptcy, which affected his credit score. And because he didn’t have the money for basic needs, he had to forgo some of his diabetes treatments and move into a friend’s basement.
While Michigan is a striking case of an automated system going wrong, it’s not the only area in which governments are using artificial-intelligence systems for decision making. Such systems are used in criminal justice, policing, child protective services, the allocation of health-care benefits, teacher evaluations, and more. In many cases, governments are hiring private companies to build these systems, Frank Pasquale said, and because the companies want to protect their intellectual property, they sometimes hide details of how the products work. “The trade-secret protection is a real problem, because you can’t even understand what the problems are,” he said.