If you’ve gone through the painstaking process of renting a new apartment in the past few years, you probably faced some sticker-shock. Vacancy rates are low, really low. And despite ever-present scaffolding, construction in many cities is still slow, as new tenants move in but few move out. The result is that in almost every major metro area, the rent is, in fact, too damn high.

Basic wisdom (which was largely established by rules governing public housing eligibility) warns a healthy bank account means that one’s housing costs shouldn’t exceed about one-third of a person’s pay. While that might be a prudent suggestion because, after all, people do have other bills and savings goals, it’s become virtually impossible to adhere to for many who live in major metro areas.

A recent report from the Joint Center for Housing Studies (JCHS) at Harvard, puts some numbers on just how bad this problem is: About half of all renters in the U.S. are using more than 30 percent of their income to cover housing costs, and about 25 percent have rent that exceeds 50 percent of their monthly pay.

It’s not just the poorest city-dwellers who are feeling the rent pressure. As prices rise, even those who make median incomes are finding that their rent eats away at a more significant portion of their pay than it once did for those in the middle class. It’s also not just the Millennial crowd: This problem is also  evident across different age groups, including Gen X and Boomers who never left the rental market, or find themselves back in it after the housing crash.

A big part of the problem is that fewer households are making the transition from renting to owning, which means more competition for limited inventory—driving rental prices up. Renters who would previously be able to qualify for mortgages are either finding that mortgage lenders are still super strict post-recession, or that there simply aren’t many homes in their price range—or both. “In normal times when homeownership was achievable you could get a starter home for between $150,000 to $250,000,” says Andrew Jakabovics, a senior director at Enterprise Community Partners, a nonprofit that focuses on affordable housing. “That segment of the market is basically dead.”

Jessica Rinaldi / Reuters

So instead, households with higher incomes and dreams of white picket fences remain in the rental market. Those households take up available units in the mid-to-high price ranges, for which they can afford to pay a premium. In fact, renters with incomes that top $75,000 are among the fastest growing group in the market, says Chris Herbert, the managing director of the JCHS. “Developers will be drawn to build the houses that provide the highest returns,” he says. That means not enough new apartments are affordable apartments that can accommodate low- and middle-income residents. Instead, high-priced luxury units get built first, pushing rents up and middle and low-income earners into apartments that are more expensive than they can afford. Sometimes this means pricing them out of cities altogether.

Pricey rent might not seem like that big of a deal, but it is. Housing accounts for the single largest expenditure for the bottom 90 percent of earners— which is pretty much everyone. While you may be able to scrape and skimp in order to decrease other bills like groceries, utilities, or extras like entertainment, housings costs are generally fixed—at least for a period of time. In other words: The more you’re burning on rent, the less there is for everything else.

And for the 11.25 million households that are considered the most burdened by housing costs—those spending more than 50 percent of their income—they aren’t just spending slightly more than half their income. Jakabovics says that the median of that group is well over the line, and what they spent on rent is closer to 70 percent of their income.

While the strain is awful for individuals and families, it’s actually also pretty bad for the economy. “We know that low-income households that are cost burdened spend considerably less on food and other essentials than affordably-housed folks in the same income bracket,” Jakabovics says. He estimates that even helping those who are most overburdened by the rent to decrease their housing expenses to only 50 percent of their income would put $56 billion back in their pockets. In turn, these households would put that money back into local economies.

But until housing policies change—allowing for more affordable housing to be built more rapidly, or mortgage companies loosen the tight reins placed on credit in the wake of the crisis—rental prices will continue to push upward, and those who can afford the increases least, will suffer most.