On Tuesday, U.S. President Barack Obama will host a summit on the refugee crisis on the sidelines of the United Nations General Assembly meeting. His administration reportedly plans to increase the number of refugees the United States will let in to 110,000 in fiscal year 2017, and he is expected to call on other world leaders to take in more refugees themselves. But if international coordination on refugees looks incredibly difficult, Obama also faces major hurdles to his preferred refugee policies at home.

Secretary of State John Kerry has reportedly briefed members of Congress about the Obama administration’s goal to increase U.S. refugee resettlements in fiscal year 2017 from the 85,000 it pledged to admit in fiscal 2016. Although it’s unclear what share of these new refugees will be from Syria, the  administration recently exceeded its goal of admitting 10,000 Syrian refugees in 2016, according to figures from the Refugee Processing Center. The moves are a response to what the United Nations has called the worst refugee crisis since World War II. The United Nations refugee agency reports that more than 65.3 million people were displaced in 2015, up from nearly 60 million in 2014.

The war in Syria has been a driving factor: Last year, Syria accounted for 4.9 million refugees, the largest share from a single country. But a string of terrorist attacks in Europe and the United States over the past year has sparked concern about the process by which refuges are vetted—even though, as Kathleen Newland of the Migration Policy Institute noted last year, among the 784,000 refugees that had by then been resettled in the United States since 9/11, only “three ... have been arrested for planning terrorist activities.” Of those, she wrote, two “were not planning an attack in the United States and the plans of the third were barely credible.” The situation has been somewhat different in Europe, where Germany saw three small-scale attacks by refugees over the summer, and where the body of one of the attackers who killed some 130 people in Paris in November was found with a Syrian passport that turned out to be fake. Following the Paris attacks, more than half of U.S. governors—all of whom were Republican except one—called on the Obama administration to halt the resettlement of Syrian refugees. The discord between Republicans and the Obama administration is evident in recent statements from lawmakers. Republican U.S. Senator Jeff Sessions dubbed the president’s latest expansion “reckless and extreme.”

Yet Obama’s current resettlement plans appear modest compared to the numbers of refugees that some other countries admit, and that the U.S. itself has admitted at other points in its history, in response to crises that didn’t match the scale of the current one.

After admitting more than 250,000 Europeans displaced by World War II, the U.S. Congress put into place the first refugee legislation in 1948. In 1975, the United States saw a wave of Vietnamese refugees following the Vietnam War, which ultimately led to the passage of a separate piece of legislation known as The Refugee Act of 1980. The law, which established the Office of Refugee Resettlement, outlined how refugees were to be admitted and resettled in the United States. It’s a rigorous process: An individual needs to fit the definition of a “refugee” as outlined by the government and be “among those refugees determined by the President to be of special humanitarian concern in the United States; not be firmly resettled in any third country; be otherwise admissible under U.S. laws.”

Since 1975, the United States has resettled more than 3 million refugees, according to the Pew Research Center. While Syrian refugees are in the news lately, the vast majority of refugees the U.S. accepted between October 2015 and May 2016 have come from Burma, followed by the Democratic Republic of the Congo and Somalia. Pew notes that the number of refugees admitted to the United States has typically been contingent on the state of conflict around the world. The Balkan wars of the 1990s helped drive the flow of resettlement in that period; it reached a peak of 142,000 refugees resettled in the U.S. in 1993.

But 2001 was a turning point in the United States. Following the 9/11 terrorist attacks, the United States passed the USA Patriot Act. As Bill Frelick, the director of Human Rights Watch’s refugee program, put it recently in World Politics Review: “[W]hen it comes to the Patriot Act’s criminalization of ‘material support’ to terrorist organizations, which was written excluding any minimal threshold: It could be a doctor providing medical care, or someone coerced—threatened at gunpoint—to cooperate with a terrorist entity.” This could disqualify any number of people in war zones, and the number of refugees admitted in the United States dipped thereafter. In fact, in 2002 and 2003, less than 30,000 refugees were accepted into the country, compared to a high of 210,000 in 1980. Later, in 2004, refugees from Somalia, Cuba, and Laos led to an uptick in admittance. The Obama administration now plans to admit 40,000 refugees from the Near East/ South Asia—including Syria—and 35,000 from Africa in fiscal year 2017, according to a Wall Street Journal report.

Yet even these numbers are well behind those of several other countries. Take Turkey, Lebanon, and Jordan, the three that have taken in the largest share of Syrian refugees. A June report from the United Nations High Commissioner for Refugees noted that “in Lebanon nearly one in five individuals is a refugee” in 2015. And, for the second year in a row, Turkey “hosted the largest number of refugees worldwide,” which amounts to 2.5 million people. Germany, meanwhile, received the most applications for asylum or refugee status worldwide—more than a million asylum-seekers are estimated to have entered the country last year.

The Obama administration has meanwhile taken some modest steps toward further opening the country’s doors to those fleeing violence, whether in the Middle East or elsewhere. The administration, along with the United Nations refugee agency, launched an initiative this year to identify people from El Salvador, Honduras, and Guatemala who are eligible for refugee status, in response to an entirely different refugee crisis that burst into American newspaper headlines two years ago as tens of thousands of unaccompanied minors flocked to the U.S. border, escaping violence in their home countries. In a July 26 White House press briefing, Eric Schultz, the principal deputy White House press secretary, said: “Our goal here to conduct the resettlement process in home countries so that we’re not dealing with an influx on the border and we’re not dealing with a heartbreaking or unsustainable situation that is neither safe or orderly.” The move highlighted a push to find ways to deal with the refugee crisis around the world more broadly. U.S. companies including Goldman Sachs, Google, and Microsoft, among others, are assisting the White House after a “call to action” by the administration.

Obama’s “Leaders’ Summit on Refugees” is the culmination of many tense months concerning how to handle a problem of staggering scale and bitter politics. If Obama hopes it will yield concrete achievements from other governments on the issue, getting such cooperation from his own Congress will also be a formidable project—and not necessarily an easier one.