Australia’s high court has rejected a challenge to the country’s practice of holding asylum-seekers at a camp on Nauru, the Pacific island nation, a decision that paves the way for the return of more than 250 people—including dozens of babies—who are now in Australia.

At issue is a case brought by the Human Rights Law Center (HRLC) on behalf of a Bangladeshi woman who entered Australia by sea. She was detained by Australian officials and taken to Nauru, which along with Manus Island, part of Papua New Guinea, is where Australia processes its asylum-seekers. The woman was returned to Australia for medical treatment during the late stages of her pregnancy, but appealed her return to Nauru. Lawyers for the woman challenged Australia’s right to detain people on foreign soil. On Wednesday, the court said the government’s actions were both legal and constitutional.

The decision paves the way for the return of about 267 people, including 37 babies born in Australia, to the detention center on Nauru.

HRLC and other immigrant-rights groups say the conditions on Nauru are poor, citing women who say they have been sexually assaulted at the detention center on the island.

“The legality is one thing, the morality is another. Ripping kids out of primary schools and sending them to be indefinitely warehoused on a tiny remote island is wrong,” Daniel Webb, the HRLC’s director of legal advocacy, said in a statement. “We now look to the Prime Minister to step in and do the right thing and let them stay so these families can start to rebuild their lives.”

Prime Minister Malcolm Turnbull said Australia’s tough stance, which is supported by the opposition, not only secures the country’s borders, but also saves lives by preventing drownings.

“Our commitment today is simply this: The people smugglers will not prevail over our sovereignty. Our borders are secure,” he told Parliament on Wednesday. “The line has to be drawn somewhere and it is drawn at our border.”

In a statement, UNICEF, the UN children’s fund, said the court’s decision shifts the international legal obligations for refugees from Australia to its poorer neighbor Nauru.

“The current offshore immigration network is a system in crisis and is creating crisis for affected children and families,” the organization said.

Under Australian law, those asylum-seekers who are granted refugee status are either settled in Nauru or Papua New Guinea—not Australia. They also have an option of going to Cambodia, under a deal worked out between the two countries.

At present, more than 1,400 people are being detained on Nauru and Manus as they await their claims to be processed. The average length of their detention is 445 days.