Jim Lewis is a 55-year-old landlord who specializes in finding tenants for hard-to-rent units in disadvantaged neighborhoods in Baltimore. On more than one occasion I watched Lewis work his magic, often with single moms he calls “Section 8 girls,” after the former name of a federal housing-voucher program. He chauffeurs them from their front door to the vacant property, whisking them straight from the car into the newly renovated home, with little time to take in the neighborhood. He opens the door to reveal a sparkling new kitchen, refreshed with upscale amenities—glossy granite countertops, stainless steel appliances—that distract from the oftentimes dilapidated houses next door. Lewis puts his strategy in simple terms: “It’s like, if you build a better mousetrap—you know how they say that?” (The landlords in this story have been given pseudonyms in order to protect their confidentiality, which was granted as a condition of their participation in my research.)

Landlords in every city are salesmen aiming to persuade potential renters, which may lead them to underplay a property’s flaws and exaggerate its strengths. But what’s happening in cities such as Baltimore is different: Landlords lure renters to disadvantaged neighborhoods, perpetuating housing segregation and limiting social mobility. Baltimore has a long history of segregation, but today, it persists within one of the very programs designed to dismantle the problem: housing vouchers.

It was hoped that Housing Choice Vouchers, previously called “Section 8,” would break up dense concentrations of poverty by subsidizing rents for poor families. But the program has fallen far short of its promise. By and large, voucher holders are not moving to areas of opportunity. They are not finding places to rent in neighborhoods that include a mix of higher-income people, the kind of move shown in recent research by Harvard economist Raj Chetty and others to have long-term positive impacts on health and economic well-being. Rather, voucher holders are now concentrated in poor neighborhoods. This is even more the case for black voucher holders, whose neighborhoods are far more segregated than those of white voucher holders. Why are these patterns of segregation being recreated under a system that was meant to undo them?

A big part of the answer lies in a middleman: the landlord. Landlords play a key role in where people find homes. Yet their role in sorting residents in and out of neighborhoods remains largely unseen.

* * *

In Baltimore, where you live has everything to do with the color of your skin. I learned this firsthand in 2011, when I was a graduate student beginning research on neighborhoods with high poverty and racial segregation, and moved into just such an area.

To begin my own housing search, I called a number on a Craigslist ad for a one-bedroom in Park Heights, a neighborhood in the northwest part of the city, and the landlord on the other end of the line asked me skeptically if I was sure I knew where Park Heights was. I convinced him to show me the converted row home later that afternoon. When I showed up, the landlord, an Air Force retiree with a head of thick, silvery hair, took one look at me and simply said, “Sweetheart, this is not a neighborhood for a girl like you.” What he meant was that this is not a neighborhood for white people. One thing was eminently clear to me even at this early stage: Landlords have enormous power to affect where people live.

I ended up spending more than a year in Park Heights, getting to know landlords and watching them show properties, do repairs, paint and repaint walls, and unclog toilets. Residents in Park Heights, as in many other neighborhoods to which voucher holders are flocking, face unimaginable poverty and violence. Life expectancy is almost 15 years lower than in affluent, white areas of the city. One in four households lives under the federal poverty line, compared to only one in six nationwide. Unemployment is rampant. Despite the drastic drop in crime across the country, the crime spike of the 1980s and 90s has only marginally abated in neighborhoods such as this one. On the day after the riots following the death of Freddie Gray a couple months ago, a man was murdered in Park Heights in an unrelated incident—the city’s 69th murder this year.

Park Heights, like so many neighborhoods across the country, reflects a legacy of years of housing policy that confined black residents to poor areas. After the Great Depression, federally-backed mortgages made it easy for whites to buy homes, but nearly impossible for blacks to do so. Banks refused to lend in predominantly black “redlined” neighborhoods, because the investment was deemed hazardous, and white neighborhood associations enacted restrictive covenants to keep blacks out. In the 60s, real-estate agents stoked fears of black incursion in Park Heights by flipping white-owned homes one at a time—a practice known as blockbusting. As whites fled in the subsequent decade, Park Heights transformed from 95 percent white and predominantly Jewish to 95 percent black.

In 1967, Otto Kerner, then the governor of Illinois, led a commission that looked into the causes of urban unrest in cities like Detroit, Newark, Los Angeles, and Chicago. The Kerner Commission’s final report revealed discrimination entrenched in federal housing policies ranging from how home loans were granted to where public housing was built. The commission warned of a nation “moving toward two societies, one black, one white—separate and unequal.” This prophecy only became more true as public housing stock aged and inner cities across the country declined. The reality today in cities such as Baltimore is not far off from this prophecy.

In the 1990s, facing levels of concentrated poverty never before seen, Henry Cisneros, the secretary of the Department of Housing and Urban Development (HUD), proposed to “end public housing as we know it.” Nationwide, blighted high-rise towers were demolished, and vouchers were used to transfer much of the burden of sheltering the poor to the private market. Out of the five million households across the country that receive some form of federal housing assistance, over half now live in privately-owned properties. By making up the difference between what a needy household can afford and the cost of a unit in the private market, the voucher was supposed to create opportunities for people to move to safer neighborhoods with better schools and more jobs. By many accounts, it hasn’t.

* * *

In Baltimore, where nearly all of the high-rise public housing was torn down in the 90s, one in five households lives under the federal poverty line and the voucher rate is one of the highest in the country. Landlords face high vacancy rates and have difficulty collecting rent from poverty-stricken tenants. Filling all of those vacancies requires that landlords play a complicated game of matching tenant characteristics—such as family size, race, voucher status, and financial risk—to property characteristics such as size, condition, and location.

As landlords manage their portfolio of properties across different types of neighborhoods, they cherry-pick the tenants they want and match them to the units they most need to fill. “The thing is, you don’t need a lot of help when it’s a good area. But in the bad area, that’s when it’s hard,” says Oscar Mayfield, a landlord who also works as a property manager. “The key is, you got to understand that everyone needs somewhere to live. There’s a tenant for every house. You’ve just got to find the right tenant.”

In other words, there is a hierarchy of tenants, just as there is a hierarchy of homes. If the landlord plays the matching game wisely, “there’s a tenant for every house.” What this means though, is that the tenants at the bottom of the social ladder are also being matched to the worst homes, in the worst neighborhoods.

In disadvantaged areas, finding and attracting tenants who are able to pay their rent reliably is no easy task. This provides an incentive for landlords to entice voucher tenants to these units, since a big chunk of their rent is paid by the housing authority. “Everybody prefers Section 8,” Mayfield explains. “It’s tough times now. If the tenant doesn’t pay the mortgage, you have to. Section 8 ensures that you going to get your money.” Since the voucher program offers steady payment (for at least of the government’s portion of the rent), many landlords have oriented their businesses towards attracting and retaining voucher holders.

Beyond steadier rent payments, having a tenant with a voucher brings another benefit. Landlords can often charge more for units occupied by voucher holders than what they would go for on the open market. The voucher program’s rent ceilings are determined by what’s called Fair Market Rent, a standard based on average rents across the entire city, so they are often higher than what a typical unit might rent for in a less-affluent neighborhood.

One landlord I met had two identical units in the same building: The third-floor apartment was rented to a family with a voucher and went for $250 more than the identical unit on the fourth. This loophole creates perverse incentives for landlords to recruit poor voucher holders to properties in disadvantaged neighborhoods.

Baltimore’s landlords have a whole toolbox of strategies to get the tenants they want to the often underwhelming properties they need them in. They range from benign sales tricks to more insidious manipulation. For example, one young, African-American landlord says that an effective tactic is to “put the tenant in a position where they are in control. So you come into a room and say, ‘What color do you want this room?’ And they feel like now it belongs to them, so it makes them want the property even more.”

Once the tenant has rented the home, a new set of issues arises. At this point, the landlord’s main concerns are to manage the tenant’s behavior in the unit, and to keep costs low by attenuating turnover. If the tenant tries to move while indebted to the landlord, she will lose her voucher, so some landlords strategically allow tenants to get behind on rent.  As the debt accrues, it gets harder and harder for the tenant to consider a move without jeopardizing his or her subsidy. The threat of voucher loss looms large. One white landlord in his thirties explains, “It’s a big threat. If there’s 16,000 people that are waiting for a voucher in Baltimore, that’s a pretty big incentive. I would be scared.”

Together, these practices help landlords keep voucher holders trapped in units that deliver the biggest profits, in some of the very neighborhoods the voucher is supposed to afford them the opportunity to escape.

* * *

To eliminate the incentive structure that rewards landlords who convince voucher holders to live in the most disadvantaged neighborhoods, there is a simple fix. The formula that calculates Fair Market Rent should be reformed to use numbers for individual neighborhoods, rather than citywide averages. Last week, the Department of Housing and Urban Development announced that it is considering a proposal to do just this. Such a remedy could save money, and that matters since currently there isn’t enough to go around: Only one in four qualified families receives housing aid.

In addition, families need to be better informed of their rights—for example, what they should expect from the landlord, and under what circumstances they can break their lease without losing their voucher. Providing mobility counseling, as a bill recently approved in Connecticut does, would help families learn about neighborhood resources such as schools, jobs, and crime when considering a move. And extending the period of time allotted to use the voucher (in Baltimore, it’s 60 days) would mean that families who don’t find an approved home in time would not be at risk of losing their voucher.

Housing policy in the U.S. has featured various attempts at dismantling poverty. In the 1930s, slums were razed and public housing was built. In the 1990s, towers were torn down and voucher subsidies came to be seen as an effective way of letting the private market remedy concentrated poverty and segregation. In theory, voucher programs should be providing opportunities for poor households to move to better neighborhoods. But in some cases they have done just the opposite. If the goal is for housing to be part of the solution to deep poverty and inequality, housing policies need to undo patterns of residential segregation—not recreate them.