Right now, it’s hard to tell in what direction Puerto Rico and the United States Virgin Islands are heading after sustaining hits from Hurricane Irma and then Hurricane Maria in September. The picture of the damage in Puerto Rico verges on the apocalyptic now, and much of the situation is currently deteriorating. Millions are still without power, fuel, food, and water are running out, hospitals are overwhelmed and mostly inoperable, and bodies are “piling up” in morgues. Struggles in the Virgin Islands are less documented, but no less real: Power is still tenuous, flooding is still rampant, most of the hospitals have been damaged or destroyed, and many roads are ruined.

Federal aid is arriving to help deal with the onslaught of disasters, but even if that aid were adequate and weren’t heavily criticized by residents for being late or ineffective, aid alone won’t lead to recovery. Existing challenges that plagued the territories before Irma and Maria multiply the effects of the storms, and must themselves be dealt with in order for recovery to be a reality. But dealing with those issues will require examining the legacies of colonialism, racism, and erratic federal policy in shaping the destinies of Puerto Rico and the Virgin Islands, and how those forces contributed to the destruction on display today.

President Trump has—if nothing else—outlined just how deeply those legacies run in his direction of the federal response. When Trump attacked San Juan mayor Carmen Yulín Cruz on Twitter last weekend, his remarks that the relief effort was hampered by “poor leadership ability by the Mayor of San Juan, and others in Puerto Rico, who are not able to get their workers to help” and that “they want everything to be done for them when it should be a community effort,” played on similar themes as an 1899 colonial governor bemoaning the lack of “Anglo-Saxon energy to face a gloomy outlook” among his black and brown constituents on the island, which had just been taken from Spain as the United States built its own empire.

When Trump expanded on his comments Tuesday, just hours before his first trip to San Juan, he paired praise to federal responders for “the great job we’ve done” with criticism of the efforts of local officials. It was a contrast that recalled the opinion of Justice Henry Billings Brown in the 1901 Downes v. Bidwell case—one of the so-called “Insular Cases” that now form the basis of governance over the U.S. territories, and helped create the status quo where residents of the territories are citizens who pay taxes and serve in the military, but have neither representation in Congress nor sovereignty. There, Brown said that for the “alien races … the administration of government and justice, according to Anglo-Saxon principles, may for a time be impossible.”

The president’s other comments have perhaps unwittingly outlined the culpability at play in Puerto Rico’s crisis. When Trump first tweeted about Maria, four days after the Category 4 made landfall on Puerto Rico, he said that: “Texas & Florida are doing great but Puerto Rico, which was already suffering from broken infrastructure & massive debt, is in deep trouble. Its old electrical grid, which was in terrible shape, was devastated. Much of the Island was destroyed, with billions of dollars owed to Wall Street and the banks which, sadly, must be dealt with.” While Trump received criticism for implying that Puerto Ricans were at fault for their own struggles or that he might be inclined to hold their feet to the bondholders’ fire while the island is still coping with the aftermath of a calamity, his diagnosis—that Maria’s catastrophic effects in Puerto Rico are tied to endemic and ongoing structural crises—was correct.

Those crises, however, are intricately tied to the nature of America’s colonial relationship with its territories, a relationship that does not begin or end with Trump. The legacy of colonialism has been a force in the immediate phase of relief, when mainlanders largely failed to realize that Puerto Ricans are U.S. citizens, and when the basic coordination and urgency shown in federal relief efforts for Hurricane Harvey in Texas failed to materialized in Puerto Rico.

It took days of lobbying by Puerto Rico Governor Ricardo Rosselló to temporarily ease the Jones Act, a maritime law that some policymakers believe makes goods in Puerto Rico more expensive, and limits aid vessels in port to American-flag ships. After the Trump administration said on September 26 that there was no need to temporarily waive the Jones Act, Rosselló said on September 27 that he’d reached out to Congress and that “we expect them to waive it.”

The administration granted the waiver the following day, allowing foreign ships to sail to Puerto Rico’s aid along with domestic ships. Some of those American-flag assets were already moving to Puerto Rico in the aftermath of the storm. The USNS Comfort, the Navy’s floating hospital, wasn’t deployed until nine days after Maria’s landfall, and has just now arrived off the Puerto Rican shore.

“Those are things that I think should’ve been in place shortly after the storm,” argued Edwin Melendez, the director of Hunter College’s Center for Puerto Rican Studies. FEMA-assisted basic disaster logistics and preparation are critical, but appear to have been largely absent in Puerto Rico. Undelivered foods and supplies have remained undelivered because there seem to have been few existing plans for getting them to Puerto Rico’s interior, and now very few ways to track and identify needy populations, since communications outside major municipalities have largely gone dark and rural roads are difficult to traverse.

But the most dire current needs in both Puerto Rico and the U.S. Virgin Islands—and the most potentially deadly problems—are staggering infrastructure losses. According to U.S. Virgin Islands Governor Kenneth Mapp, “this will be the fifth time the U.S. government has basically paid to rebuild our power infrastructure,” after powerful hurricanes over the past three decades have destroyed homes and grids on Saint Croix, Saint Thomas, and Saint John. Mapp hopes to work with FEMA and the Department of the Interior to build hurricane-resilient power lines, but that project will take time and a generous appropriation of funds from a Congress that has not usually been generous to the territories.

Puerto Rico’s power infrastructure is also notoriously flimsy, even on good days, and one of the major components of its debt crisis is the $10 billion debt of the Puerto Rico Electric Power Authority (PREPA), which has had to run up massive credit lines just to keep its outdated, mostly oil-powered grid running, with payments from the island and its citizens always coming up short. That grid was destroyed entirely in Maria, and in Melendez’s view, “restoring the electrical system might be a longer-term project.”

That project will be difficult. The collapse of PREPA and other utilities on the island has been influenced by Congress’s decisions to end corporate-friendly tax codes and subsidies, and the Puerto Rican economy has suffered from depression and inefficiency over the past two decades. The remedy from Congress, the Puerto Rico Oversight, Management, and Economic Stability Act (PROMESA), did help fend off the worst of the debt crisis, but also installed a congressionally-appointed board of directors who can basically overrule the Puerto Rican government and has thus destroyed any semblance of self-determination. While last week Puerto Rican officials turned down a further $1 billion loan from PREPA’s creditors that would have probably dragged the island’s finances even lower, just how the new electrical system for the entire territory will be built is anyone’s guess.

The debt crisis in Puerto Rico, and less catastrophic financial troubles in the Virgin Islands, have helped highlight other infrastructure issues beyond the power grid as well. Puerto Rico was suffering from humanitarian and public-health crises, including zika outbreaks, shortages of physicians and prescriptions, and lack of health-care access in rural areas when Maria hit as well, and even before the storm hospitals suffered rolling blackouts, prescription shortages, and the possibility of permanent disconnection from utilities. Part of those problems were due to a completely arbitrary system of funding for Medicaid, which saw Puerto Rico receiving fewer dollars per Medicaid enrollee than the states, and also saw the Puerto Rican government on the hook for almost half that amount. Although the territories were granted a reprieve in the form of lump Medicaid payments included in the Affordable Care Act, those funds are temporary and weren’t capable of keeping up with previous health-care issues, let alone the public-health nightmare now settling on the islands after Maria.

“Medicaid is one of the biggest expenses in Puerto Rico, the health system is in shambles right now, and we still have to pay 50 percent of total expenditures,” Melendez says. “Puerto Rico’s broke, so they can’t borrow that money anymore. If the ACA program is not attended to we’re gonna be done.”

Similar problems abound in the Virgin Islands. According to Mapp, the U.S. Virgin Islands completely lost two hospitals during the hurricanes, and the roofs and critical components of other hospitals were damaged. But since the majority of hospital revenues come from Medicaid, it’s difficult to rebuild the hospitals while also paying for the immediate needs of people injured or sickened by the flood, who often now have to be airlifted to the mainland. Mapp says that he is working with the Department of Health and Human Services to potentially waive cost-sharing from the territory on Medicaid for up to 18 months.

As officials on the U.S. Virgin Islands and Puerto Rico continue the months-long work of stabilizing their territories in the immediate aftermath of the two hurricanes, they are also awaiting the congressional aid package that will signal if recovery will proceed.

On the one hand, if Republicans in Congress follow Trump’s lead on holding territorial citizens to their debts and passing the buck to them, the resulting package could result in death sentences. Puerto Rico is facing a mass exodus of young residents to the mainland as its economy collapses and as jobs dry up, and those losses would only continue if Puerto Rico’s bondholders continue their stranglehold on the island. The Virgin Islands’ situation is less complicated, but it’s also facing significant emigration, and policies that stop short of committed rebuilding programs could weaken tourism and the construction industries enough for people to leave to the mainland, too. And health-care expenses could easily spiral out of control, especially if hospitals aren’t rebuilt on time and airlift continues to be the standard practice.

On the other hand, Congress—which has almost complete control over the legal entities of the territories—now finds itself in position to reverse some of the measures that have eroded the status of territorial citizens and made local economies increasingly untenable. Congress could vote to subsidize or forgive some or all of Puerto Rico’s debt, it could authorize and expand mass community-development block-grant programs to build resilient housing in places that are now ruins, and it could finally act to create a comprehensive medical system in the territories that doesn’t just aim for funding parity between citizens there and citizens in the states, but invests in reversing the unique public-health problems that have arisen as a result of second-class citizenship. And perhaps Congress could make that strategy official policy across the board in its response, investing to right wrongs and help territorial citizens create resilient communities that will fare better in the next storm and the next. Perhaps, also, Congress could relinquish its absolute control.

Those concerns still seem lifetimes away. Final aid packages and insurance statements probably won’t be available until the storm-related damages can be reported, an exercise that can’t yet happen in most places in the interior of Puerto Rico. People are still dying, families still haven’t made contact with loved ones, diseases are setting in, and food and water are still running out in places. Doctors in both Caribbean territories are performing life-saving work in tents with generators. But these all signal that the recovery will be an incredibly long process. And for the sake of the territories, the government that brought them in as colonies will have to undo some of the choices and policies made in the name of dominion in order to complete that recovery.