DeShawn Smith lived, for awhile, with her partner and two children in a good neighborhood in the Texas town of Beaumont. Her kids loved books and reading, and didn’t want summer time to come, because it meant the end of school. Then Smith’s partner went to jail, and everything changed.

Without his income, she could no longer afford the apartment in the nice part of town. She moved to a public-housing property in a highly-segregated neighborhood, next to a cement-crushing plant. The ceiling leaks and trains rattle by all night, and the bathtub is caked with mildew. Her daughter, who is now 8, hates her new school, and said her teacher confessed that she only came to school for a paycheck. The same teacher told Smith that her daughter was the only second-grader in the class who knew how to read.

Smith hates what the move has meant for her children and their education. But the old apartment was too expensive, and “I couldn’t afford it by myself anymore,” she told me, a few months ago, when I was in Beaumont.

America spends $80 billion a year incarcerating 2.4 million people. That money is spent on things like beds, staff, food, and facilities—but the effects of that incarceration are costly in other ways too, according to a new report out Tuesday from the Ella Baker Center for Human Rights. The report is a result of a yearlong project that collected information in 14 states through focus groups and surveys (My colleague Ta-Nehisi Coates writes extensively about mass incarceration in his October cover story).

When a someone goes to prison, nearly 65 percent of families are suddenly unable to pay for basic needs such as food and housing, the report found. About 70 percent of those families are caring for children under the age of 18. Women like Smith are often responsible for court-related costs associated with the conviction, and many families go into debt to pay those fees, leaving even less for food and shelter. When that family member gets out of jail, their loved ones are left with the task of supporting their reentry. This burden is ongoing since people with a criminal record often are unable to find work upon their release.

“Poverty, in particular, perpetuates the cycle of incarceration, while incarceration itself leads to greater poverty,” the report says.

Going to prison is an increasingly costly proposition. In 40 states, people facing imprisonment must find the money for things such as attorney fees, court fees, bond and restitution—the total costs of which average about $13,607. Many defendants choose to pay for private attorneys, but even those who don’t must pay application fees for public defenders in 43 states, which can range from $10 to $480.

Fees associated with criminal justice have become a big revenue generator for states, but saddle incarcerated people and their families with significant debt: Now, up to 85 percent of people returning from prison have some sort of criminal-justice debt, up from 25 percent in 1991, according to the report.

The costs don’t stop after a trial and conviction. Families who try to maintain contact with a prisoner must pay high fees for phone calls from prison. The Prison Policy Initiative found that families pay $1 billion annually to call relatives in prison, and until 2013, calls could cost $17 for a 15 minutes (the FCC cracked down on this and those calls now cost $3.75). Visiting prisons is also costly for families, who must pay for background checks in many states, as well as transportation to the facility. One-third of families in the report said they went into debt to pay for visitation or phone calls.

And when—and if—the individual is released from prison, the costs continue to mount. Three in four people surveyed found it difficult to find work after being released from prison, and only 40 percent of those surveyed were working full-time after five years. This makes it difficult for one-time prisoners to pay child support, or contribute to housing or food costs. It also makes it difficult for them to further their education, one of the only things that can improve their financial prospects.

Those released can also be denied public housing because of their conviction history, and their families can be kicked out of public housing if they have someone living with them who has a criminal record. Nearly 80 percent of those surveyed said they were ineligible for or denied housing because of their criminal record.

“All of the places that I wanted to live—that were nice and where I could raise kids told me ‘no.’ So I ended up where I am now, in a rundown four-plex that’s a slum with moldy walls,” one focus-group participant in Kansas told the interviewers.

The Baker Center does have a few recommendations for how to improve life for prisoners and their families. The first centers on restructuring criminal-justice policies. States could follow the lead of California, which in 2014 reclassified six non-violent offenses from felony charges to misdemeanors in order to reduce prison time. States and cities could also look into alternatives to incarceration, including programs that require people who commit crimes to be held accountable by those harmed.

The second recommendation concerns the barriers that exist when people get out of prison. HUD and other public-housing entities could make it easier for people with criminal records to be eligible for public housing. States and cities could remove the section on job applications that requires people to disclose a criminal history. And they could also rethink current policies that make people with criminal records ineligible for Pell Grants and other federal programs that make it easier to get an education.

Finally, the report recommends that communities reverse budget cuts and give more on-the-job training and reentry preparation to prisoners.“We can no longer afford to continue business as usual with criminal justice in the U.S.,” the report concludes. “If our nation truly wants to support all families and communities … we must drastically reduce the financial, emotional, and health impacts on incarcerated people and their loved ones.”