In 2008, a small, conservative lobbying firm know as Citizens United wanted to show a documentary about Hillary Clinton on broadcast television close to the Democratic primary. The D.C. District Court ruled that this was a violation of a 2002 law commonly known as the McCain-Feingold Act, which prohibited corporations and unions from funding “electioneering communications” about a political candidate within 30 days of a primary or 60 days of a general election. Two years later, the Supreme Court overturned that decision in the case Citizens United v. FEC, ruling that corporations, including those that are for-profit, can spend unlimited amounts of money on “electioneering communications,” including television advertisements.

Over the past five years, this decision has had a major effect on American politics, but the biggest changes haven’t been at the presidential level. That’s the argument advanced by David Boies, a prominent attorney, at a session at the Aspen Ideas Festival on Wednesday. “The real problem is at the state and local level, where an infusion of a certain amount of money can totally swamp [elections],” he said. Although the decision didn’t alter the limits on direct donations to candidates’ campaigns, it facilitated the creation of what are now known as Super PACs, organizations that can spend unlimited amounts of money to support or oppose candidates as long as they don’t coordinate directly with campaigns.

As Norm Ornstein wrote for The Atlantic last October, this has had a huge effect on judicial elections in particular:

Loads of money—mostly conservative—went into judicial-retention elections in the last cycle in Florida, following a similar experience in 2010 in Iowa and Illinois. We saw similar efforts on a smaller scale in other states, including Wisconsin and Michigan. All had a ton of attack ads. Those efforts have exploded in the 2014 elections. In North Carolina, where repeal of the state's Judicial Campaign Reform Act by the right-wing legislature opened the door to a further explosion of campaign spending, and where the GOP sees retaining a majority on the court (ostensibly, but risibly, nonpartisan) as a key to their continued hegemony in politics, the Republican State Leadership Committee spent $900,000 on an unsuccessful primary campaign to unseat Justice Robin Hudson, and will target Court of Appeals Judge Sam Ervin IV in his second attempt to move to the Supreme Court (the first one, in 2012, cost $4.5 million or more).

In April, the Supreme Court took up a narrow question related to judicial-election donations in Williams-Yulee v. Florida State Bar, ruling that candidates themselves cannot ask for money. But, Chief Justice John Roberts added, future challenges will be examined under “strict scrutiny,” which means that it may be more difficult for regulators to place further restrictions on this kind of election spending.

As Justice Ruth Bader Ginsburg wrote in her partial concurrence to the Court’s opinion:

In recent years ... issue-oriented organizations and political action committees have spent millions of dollars opposing the reelection of judges whose decisions do not toe a party line or are alleged to be out of step with public opinion. ... [H]uge amounts have been spent on advertisements opposing retention of judges because they rendered unpopular decisions in favor of criminal defendants.

In the session in Aspen, Boies debated the effects of Citizens United with his colleague Theodore Olson, who famously opposed him in the 2000 Supreme Court case Bush v. Gore but served as his co-counsel in Hollingsworth v. Perry, the 2013 Supreme Court case that overturned California’s Proposition 8 ballot initiative. While Boies characterized Citizens United as a decision that stifles democratic participation, Olson, who represented the plaintiffs in the case, argued that it’s a defense of free speech.

“The more speech we have, the better—that’s what the Framers of the Constitution thought,” he said. One of the key disagreements in Citizens United is whether money counts as speech—the Court accepted Olson’s argument that it does. “It might be nasty speech, it might be unpleasant speech it might make you uncomfortable. The answer to that is the marketplace of ideas.”

But Boies argued that the Supreme Court mischaracterized the effect that money has on politics. In its opinion, he said, the Court argued that there’s a danger of corruption “with respect with contributions to political candidates, but there is less of a danger with regard to independent expenditures. Who knows that? That’s not something that courts are well-designed to determine.” The Court’s argument follows, he said, if you believe that making political donations is the same as making political statements, but “if you believe that speech and money are different … that money enables speech, but is not speech itself, and if you believe that people really are different [from corporations], then the syllogism breaks down.”

As they asked questions at the end of the session, many members of the audience seemed to be fairly opposed to the decision. But at least one member of the audience likely came to the conversation with a different perspective. David Koch—whose organization, the Koch Family Foundation, consistently spends large amounts of money in support of conservative candidates for office—was sitting in the front row.