Ruinous floodwaters, pipe-bursting cold, and destructive wind do not discriminate when it comes to victims. But while natural disasters may strike without prejudice, the path to recovering from them is much less equal.

To start, natural disasters displace a tremendous number of people every year. A new report from the Norwegian Refugee Council and the Internal Displacement Monitoring Centre finds that in 2014, more than 19 million people around the world were forced from their homes due to natural disasters. On average, over 22 million people have been displaced each year since 2008 because of climate or weather-related disasters. And rising sea levels, sudden temperature swings and other problems hint that things will only get worse; around the globe, the chance of being displaced by a natural disaster is 60 percent higher today than it was in the 1970s.

While a tragedy that leaves hundreds of thousands adrift might receive lots of attention in the moment, interest and aid are often fleeting. That’s especially the case for the many families unable to rebuild or return to their homes, considering that the researchers turned up evidence of victims who had been displaced after a disaster for as long as 26 years.

Major damage and displacement due to natural disasters disproportionately affect those living in the developing world, where such occurrences are far more common (because of less developed infrastructure and emergency-response plans). But helping and rehousing natural-disaster victims can be just as difficult in the U.S. In rich countries, the most vulnerable groups are still the ones who suffer the most in the aftermath of a natural disaster.

Hurricane Katrina is an illuminating example. In 2005, it displaced more than one million residents of the Gulf Coast. Once rescued from their flooded homes, many were shipped off to neighboring states where they had no family, no job, and little idea of how long they would be gone.

A decade later, some still are unable to return home. Some haven’t returned by choice, since they’ve already made new homes elsewhere. But there are also many who would prefer to go back, if they could. One thing stopping them is that moving, rebuilding, and elevating homes in order to comply with new post-Katrina safety standards costs money that many poorer residents simply don’t have. With a lack of affordable rental housing in the city, the many tenants booted from their homes also have few options if or when they return.

Such problems extend beyond New Orleans. After Hurricane Sandy swept up the east coast, officials estimated that recovery efforts would take about two years. Almost three years later, the internal displacement report finds that 39,000 people in affected areas—which range from the Appalachians to New England—were still in need of permanent housing or assistance with housing.

In a blog post, Elizabeth Ferris, a senior fellow at the Brookings Institution wrote,

Those who are poor and marginalized often suffer disproportionately from the effects of disasters, in part because they tend to live on marginal land and their houses are more weakly constructed. They are also less likely to own their homes, which means that it is less likely they are eligible for assistance to rebuild.

A recent report from the Center for American Progress expounds upon this idea of unequal impact, noting that low-income housing is poorly equipped to cope with the impact of extreme weather because it is often old, poorly maintained, or shabbily built. It’s true of the Lower Ninth Ward, which was ravaged by Katrina. It’s true of Galveston Island, Texas, where hundreds of low-income housing units were destroyed by Hurricane Ike. And it’s true of New York City, where nearly half of the city’s 40,000 public-housing residents were displaced by Sandy. When affordable housing is deeply damaged in places such as these, it often takes years to replace—leaving residents to flounder in the meantime.

The problems, though, aren’t just the inconvenience, stress, and uncertainty that come with displacement. It’s also the economic toll. For some, it’s the cost of paying rent while also paying the mortgage on an uninhabitable property. For others, it’s the burden of paying out of pocket for essentials—like a bed, or car—while waiting for reimbursement from aid funds. Meanwhile, the cost of fortifying a property so that the next storm, flood, or instance of high winds doesn’t do the same amount of damage, can be prohibitively high.  

Government-assistance programs can go a long way in helping the displaced, but they’re sometimes structured such that the neediest groups have a difficult time receiving help, or are left out. Part of the problem is the gulf  between victims and the organizations tasked with helping them. Many Sandy victims said they hadn’t received relief dollars up to two and a half years after the event, and hadn’t been told what was causing the delay. Others were unaware of what type of aid was available, or if they qualified. On top of all that, the information about relief programs that did exist  wasn’t immediately available in other languages, specifically Spanish—a big problem for a segment of the country with such a large immigration population.

Another group that can be overlooked by aid programs is renters, who tend to be poorer, and minorities. While they may not own the destroyed building that was once their home, renters lose a lot of their personal property in disasters. During Sandy, 40 percent of the homes damaged in New Jersey, the hardest-hit state, were renter-occupied, but tenants collectively received only 25 percent of the financial assistance assigned to the state as of 2014. (This aid program has since been modified and starting in 2015, a larger portion of aid, 33 percent, has been directed towards renters.)

In the wake of a hurricane, tornado, or earthquake, there are inevitably scenes of a community coming together to rebuild. Local officials, neighbors, and even strangers from far-flung destinations show up to pitch in. Such unity is important, but it’s also largely short-lived. This often means that months and years after a community has supposedly “recovered, the neediest members are still struggling to find their way home.