Perhaps not since the collapse of the Soviet Union has there been such a vogue for tearing down statues. And just as the removal of images of Lenin and Stalin rubbed nerves across the former Soviet Socialist Republics, the effacing of statues in the United States has become an acrimonious debate.

The most recent flashpoint came in New Orleans, where Mayor Mitch Landrieu ordered the removal of statues of several Confederate generals. In the face of massive protests, Landrieu was forced to resort to both heavy police presence and unannounced nighttime removal to get the statues down. But there are plenty of other examples, beginning with South Carolina’s decision to quit flying the Confederate battle flag at the state capitol and running through more recent skirmishes from St. Louis to Charlottesville, Virginia.

Some of those who oppose the removal of those statues are racist—some proudly so, others less openly. (One need only read quotes from those rallying at sites in New Orleans, or the white supremacists organizing in Charlottesville, to see this.) But others bring different objections to the debate. Some objections, for example, are borne out of a sense that removing monuments is a rebuke to the memory of ancestors who fought bravely for the Confederacy. And a final group sees some wisdom in removing some statues, but worries about a slippery slope—foreseeing a future in which even statues of George Washington and Thomas Jefferson are torn down.

The first group cannot, and should not, be accommodated. But the others raise questions that are at least worth answering, and a panel Wednesday at the Aspen Ideas Festival, co-presented by Aspen Institute and The Atlantic, wrestled with how to do that.

Landrieu, whose speech justifying the removal became an improbable viral sensation, confessed that he hadn’t anticipated the level of backlash that removing the statues would cause. He figured that the more than 150 years since the war ended would have provided sufficient distance—but he was wrong.

“I thought it was really important to say that the Confederacy lost,” he said, “The cause of the south was not a just cause.”

The idea that taking down the monuments dishonored the Confederate dead is one commonly refuted. The statues being removed were not in cemeteries, nor were they dedicated to the Confederate dead—but to specific men. But even that was a charade. As Landrieu pointed out, many of today’s contested Confederate monuments were raised long after the war, during periods of white backlash against civil rights: in the Redemption period, or during the mid-20th century civil-rights movement.

“They were not statues that were put up to honor those particular men,” he said. “It was to send a message that the Confederacy was really the right cause, and not the wrong cause.”

The Lost Cause narrative was allowed to take root in part because of a desire among whites in both the North and South to foster reconciliation.

“The idea was to bring the country back together, and that’s what whites did,” said Annette Gordon-Reed, the Harvard professor of American history. “It’s like beating up your little brother and then you feel guilty and then you let him have his way.”

Viewed in this light, the removal of monuments is not an erasure of history, but an effort to revise the popular account toward a more accurate one: “We’re just beginning, in the last 40 years, to tell the true story of the country,” she said.

Some hesitation about removing monuments is grounded in a sense among Southerners of still being condescended to, Landrieu said. “I think some of the pushback is [the sense that] if we admit this and we admit we were wrong, it will feed into the misapprehension that people have” about continued racism in the South. Of course, it is just the opposite—the backlash only brings unwanted attention to the persistence of Confederate monuments—but as the poet Elizabeth Alexander pointed out, the North is hardly immune to racism itself. “As Bree Newsome said,” referring to the activist who, as part of a protest removed the Confederate flag from the South Carolina state house grounds in 2015, “the Confederacy may be a southern issue, but white supremacy is an American issue,” Alexander said.

Nevertheless, the concerns about erasure of history remain perhaps the most potent objection, espoused not only by irredentist rebels but even by those who declare strong disdain for the Confederacy. And Gordon-Reed offered two rejoinders.

The first was that removing a statue hardly constitutes erasing history. “We’re always going to know who Robert E. Lee is,” she said. “The question is where these monuments are. The public sphere should be comfortable for everybody.”

But what about the idea that once the Lees and Stonewall Jacksons and P.G.T. Beauregards are pulled down, the revisionists will inevitably start agitating for pulling down monuments to slave-owning Founding Fathers like George Washington and Thomas Jefferson.

But Gordon-Reed, who won the Pulitzer Prize for her book on the relationship between Thomas Jefferson and Sally Hemings, said it was not hard to draw a bright line separating Jefferson’s generation of Virginians from the ones who tried to secede.

“We can distinguish between people who wanted to build the United States of America and people who wanted to destroy it,” she said. “It’s possible to recognize people’s contributions at the same time as recognizing their flaws.”

“You’re not going to have American history without Jefferson,” Gordon-Reed said. Alluding not to the demise of the Lenin statues but to the infamous deletion of disgraced figures from Kremlin photographs, she added, “It’s not the Soviet Union.”