‘The Greatest Talker of His Time’

Felix Frankfurter was an eloquent liberal champion of judicial restraint. Is it time for a reappraisal?

sketch of Felix Frankfurter wearing judicial robes and glasses
Illustration by Oliver Munday. Source: Photoquest / Getty.

In September 1953, with the Supreme Court only months away from rehearing oral argument in Brown v. Board of Education, Justice Felix Frankfurter received word while vacationing in Massachusetts that Chief Justice Fred Vinson had died suddenly of a heart attack. Returning to Washington so that he could attend Vinson’s funeral, Frankfurter bumped into his former law clerk Philip Elman in Union Station. Frankfurter did not exactly appear staggered by grief. To the contrary, Elman observed the 5-foot-5 Frankfurter walking with a particular spring in his diminutive step. Vinson’s unexpected departure might enable the Court to issue an effective decision outlawing racial segregation in public schools, an outcome that was, Frankfurter believed, well beyond the late chief justice’s meager intellectual and leadership capabilities. Frankfurter gripped Elman by the arm, stared at him intently, and uncorked the following line: “Phil, this is the first solid piece of evidence I’ve ever had that there really is a God.”

This yarn encapsulates vintage Frankfurter in at least two distinct senses. First, he was a lively, often dazzling conversationalist. Despite his never having heard a word of English before he immigrated to Manhattan from Vienna at age 11, elite Americans widely celebrated his silver tongue. As The New York Times would memorialize Frankfurter in 1965, “He was … bursting with joy and wit and sarcasm, eager to exchange gossip or debate eternal verities—but at any rate, to talk. He was by all odds the greatest talker of his time.” This encomium was not one that Frankfurter received only posthumously. A 1960 book titled Felix Frankfurter Reminisces, which consisted merely of transcribed interviews, became an improbable best seller and a finalist for the National Book Award. That volume forthrightly conceded that it was not a full-scale autobiography—indeed, it did not even broach Frankfurter’s years as a justice—but rather was “just talk.” Still, the response was rapturous.

Magazine Cover image

Explore the September 2022 Issue

Check out more from this issue and find your next story to read.

View More

Second, the vignette illuminates Frankfurter’s preternatural penchant for spotting, cultivating, and advancing talented young men. Elman was not just a former law clerk; he was then working in the solicitor general’s office at the Department of Justice, where he’d assumed primary responsibility for drafting the government’s briefs regarding school segregation. Frankfurter communicated with Elman about Brown constantly, helping shape the government’s arguments. Elman noted, without any hint of rancor, that Frankfurter treated him “not as a lawyer in the SG’s Office, but as his law clerk for life.”

Elman was only one of many Frankfurter clerks who went on to assume influential positions in national life. Others included the Washington Post publisher Philip Graham, Attorney General Elliot Richardson, and Secretary of Transportation William T. Coleman, whom Frankfurter had hired as the Court’s first Black law clerk in the 1940s. (Frankfurter’s egalitarianism did not, alas, extend to gender; he pointedly refused to hire a young attorney named Ruth Bader Ginsburg even though she received glowing endorsements from several of Frankfurter’s usual sources.) And before becoming Justice Frankfurter, Professor Frankfurter had helped supply the federal government with brainpower by dispatching his sharpest Harvard Law School students to Washington. Frankfurter, who had no children of his own, hazed and doted on these protégés in equal measure, and in the process inspired a lifetime of fealty.

In one conspicuous sense, though, the bon mot—or perhaps mal mot—elicited by Vinson’s death clashes with the dominant perception of Frankfurter. Recall that Frankfurter’s jubilation was driven by hope that the Supreme Court would exercise its authority to invalidate school segregation. Yet Frankfurter was, throughout his time as a justice, the nation’s preeminent advocate of judicial restraint. The mighty power to deem laws unconstitutional in a democratic society, he believed, should be exercised only in the most glaring, egregious circumstances. When a group of nine lawyers possessing lifelong appointments vetoes actions taken by elected officials, democracy itself is typically the loser.

Concerns about the judiciary abusing its review authority were not mere abstractions for Frankfurter. During his second year as a law student at Harvard, the Supreme Court used the Fourteenth Amendment’s due-process clause to invalidate legislation designed to protect employees in an infamous case called Lochner v. New York. This decision from 1905 became Frankfurter’s “Rosebud” moment. During the next few decades, the Supreme Court continually wielded the Constitution to strike down progressive economic policies, most saliently during the New Deal. Frankfurter, like many other legal liberals, defined himself against these Lochner-era usurpations, vowing that if he ever ascended to the bench, he would delineate a modest role for the judicial branch. “The real battles of liberalism are not won in the Supreme Court,” Frankfurter wrote in a New Republic article in 1925. After being confirmed as an associate justice in 1939, Frankfurter honored his vow, self-consciously positioning himself as the inheritor of Justice Oliver Wendell Holmes’s legacy of judicial restraint. Holmes had been his friend, mentor, and hero, and Frankfurter seldom tired of invoking the great jurist.

Few Supreme Court justices have ever experienced steeper declines in reputation than Frankfurter. Peter Edelman, an exceptionally bright Harvard Law student in the late 1950s, recollected that in Cambridge during that era, “Felix Frankfurter was God.” By 2005, however, one legal scholar spoke for many in labeling Frankfurter “an overrated judge who left a very limited judicial legacy.”

Much of this reputational free fall is explained by Frankfurter over time becoming a man with no country. For liberals, he offered an emaciated conception of the judiciary’s responsibility for providing protections to marginalized groups. His dissents from progressive constitutional victories of the mid-20th century began to mark him as a jurist from a bygone age, one still feverishly waging the wars of yesteryear. For conservatives, Frankfurter’s commitment to advancing the causes of liberalism before he took his seat on the bench—including helping found the ACLU, staff New Deal agencies, launch the New Republic, and defend Sacco and Vanzetti—identified him as a deeply suspect political figure. In the legal sphere, too, more and more conservatives have in recent decades abandoned even the veneer of judicial restraint. Judicial engagement has become the operative term, as the right has successfully demanded that the Supreme Court exert its authority to invalidate laws involving firearms, campaign finance, and voting rights.

Brad Snyder’s comprehensive, compelling, and generally admiring biography—Democratic Justice: Felix Frankfurter, the Supreme Court, and the Making of the Liberal Establishment—arrives at a moment when the justice’s stock may, in some quarters, seem poised for a rebound. Thanks in no small part to Senator Mitch McConnell and President Donald Trump, six Republican-appointed justices now sit alongside three Democratic-appointed justices on the high court. Today, in the aftermath of Dobbs v. Jackson Women’s Health Organization, many liberals fear that the Roberts Court could even someday strike down laws in blue states permitting women to obtain abortions. On the horizon, the Court seems poised to invalidate affirmative-action plans and other policies esteemed on the left. It is no wonder, then, that several prominent left-leaning academics have recently begun advocating for a dramatically diminished role for jurists. The judicial strides toward equality that occurred during the mid-20th century were historical anomalies, these scholars insist, and they have come to assert—echoing Frankfurter—that liberalism’s real battles cannot and should not be fought in the Supreme Court. Might Felix Frankfurter, acolyte of judicial restraint, serve as the intellectual poster boy for this renewed age of judicial skepticism?

Snyder challenges conventional assessments of Frankfurter by skillfully placing him into the rich, changing context of American liberalism during the first six decades of the 20th century. It is misguided, Snyder suggests, to view Frankfurter as a wild-eyed leftist during his pre-Court career who suddenly transformed into a raging reactionary upon confirmation. This wrongheaded perception overlooks that Frankfurter—inspired by the foundational scholarship of Harvard Law School’s James Bradley Thayer—consistently cautioned against permitting the judiciary to occupy an outsize role in American life. What judges give with one hand, they can just as readily take with the other.

Snyder’s portrait of Frankfurter certainly cannot be accused of concealing the justice’s rather substantial warts. Snyder does, however, cast him in a flattering light, depicting the justice largely as he depicted himself—as a champion of democracy, and therefore an opponent of juristocracy. “In contrast to many of his [judicial] colleagues, Frankfurter insisted that the best way to protect people’s rights was through the democratic political process,” Snyder writes in his spirited epilogue. “He understood that nothing was more damaging to our democracy than to expect the Supreme Court to solve our problems … We the People needed Felix Frankfurter to steer generations of elite lawyers into public service, to shape the liberal establishment, and to oppose government-by-judiciary.” Frankfurter would not only approve of Snyder’s concluding message; he would cherish it.

As Snyder notes, moreover, Frankfurter’s invocation of judicial restraint was not an unyielding absolute. Frankfurter evinced particular concern about states’ efforts to subordinate Black citizens, and he sometimes supported using the Reconstruction Amendments to set aside such schemes. Regarding Brown v. Board of Education, Frankfurter is in legal circles widely known—and widely reviled—for persuading Chief Justice Earl Warren to insert four notorious words during the remedy stage of the decision. Desegregation should unfold “with all deliberate speed,” Warren wrote at Frankfurter’s behest, and that phrase would eventually be viewed as having blessed the South’s massive resistance. Yet even if this terminology was unwise and infelicitous, it should not bear a disproportionately large share of the blame for southern recalcitrance. The phrase also should not be permitted, Snyder contends, to obscure Frankfurter’s indispensable role in helping Warren achieve unanimity in the momentous school decision.

Still, even amid a resurgence of judicial skepticism, more than a few roadblocks obstruct the path to any potential Frankfurter revival. Consider only some of the many available instances when reliance on judicial restraint curdled into judicial abdication, as he refused to check repugnant governmental actions. In the 1930s, a public school board in Minersville, Pennsylvania, expelled students who were Jehovah’s Witnesses for refusing to recite the Pledge of Allegiance on account of their religious commitments. Although Frankfurter harbored reservations regarding the wisdom of such expulsions, he nevertheless wrote an opinion for the Court in 1940 deeming it permissible for educators to punish the pupils. “The wisdom of training children in patriotic impulses … is not for our independent judgment,” he intoned.

This misbegotten decision provoked violent attacks against Jehovah’s Witnesses across the nation and elicited scorn from journalists and academics alike. After the outcry, some justices changed their mind about the constitutionality of laws requiring the pledge, and the Court in 1943 reversed course to ban these mandates in West Virginia State Board of Education v. Barnette. Frankfurter, however, remained resolute, publishing a vehement, lengthy dissent contending that the judiciary had overstepped its bounds. Despite the dissent’s passion and prolixity, he offered no effective response to Justice Robert Jackson’s magisterial majority opinion in Barnette, which took dead aim at Frankfurter’s unduly broad conception of judicial restraint. “The very purpose of a Bill of Rights was to withdraw certain subjects from the vicissitudes of political controversy, to place them beyond the reach of majorities and officials and to establish them as legal principles to be applied by the courts,” Jackson explained. “One’s right to life, liberty, and property, to free speech, a free press, freedom of worship and assembly, and other fundamental rights may not be submitted to vote; they depend on the outcome of no elections.”

In 1944, one year after Barnette, Frankfurter issued an even uglier concurring opinion in Korematsu v. United States. That case evaluated the government’s decision to consign all persons of Japanese ancestry living on the West Coast—including many U.S. citizens—to internment camps, which the government branded “Assembly Centers.” Although this policy evicted citizens from both their homes and their lives despite no evidence of impropriety, Frankfurter refused to deem it unconstitutional, because he wished to avoid intruding into the affairs of Congress and the president. “That is their business, not ours,” he claimed. It may be tempting to believe that—during the height of World War II, not long after Japan’s attack on Pearl Harbor—no judge would have condemned what is, by modern lights, an obviously bigoted policy. Yet in his dissent, Justice Frank Murphy correctly attacked the very governmental actions that Frankfurter refused to disavow, calling them a decline into “the ugly abyss of racism,” and disparaging Korematsu as a “legalization of racism.”

Barnette and Korematsu arose early in Frankfurter’s tenure, but judicial restraint remained his guiding light up through his final opinion, a dissent, in 1962. In Baker v. Carr, the Supreme Court issued a decision that paved the way for the one-person, one-vote principle, finding that Tennessee’s refusal to reapportion its wildly unequal districts presented a problem that the Constitution could in fact resolve. Frankfurter, singing one last time from the restraint songbook, excoriated his colleagues for blithely strolling into a political morass. “In a democratic society like ours,” Frankfurter stated, “relief must come through an aroused popular conscience that sears the conscience of the people’s representatives.” Although Frankfurter fancied himself the ultimate democrat, many scholars have persuasively argued that Baker v. Carr and its progeny can be understood as buttressing our democratic order. Democracy is a famously protean term, and Frankfurter too often failed to appreciate that the judiciary’s invalidation of laws can support democratic values rather than subvert them.

Apart from such discomfiting votes, his frailties regarding judicial collegiality and judicial writing seem likely to dim the prospect of a Frankfurter renaissance. He habitually treated his intimates as either demigods to be worshipped or disciples to be tutored—which made him a wonderful surrogate son to powerful men and an engaged father figure to legal whiz kids. But he struggled profoundly to maintain close relationships with his brethren. Frankfurter often viewed his fellow jurists not as peers, but as dull-witted first-year law students. In an effort to rally Justice Stanley Reed to his position in one case, he noted: “It is the lot of professors to be often misunderstood by pupils … So let me begin again.” Legend held that if Frankfurter felt strongly about the disposition of a case, he would lecture his colleagues for 50 minutes, the standard length of a Harvard class. No matter how bravura the substance of his disquisitions, this act, predictably, grew tiresome. “All Frankfurter does is talk, talk, talk,” Chief Justice Warren grumbled. “He drives you crazy.”

On no subject did Frankfurter expend greater energy than reminding anyone within earshot of his closeness to and affection for Justice Holmes. As Justice William Brennan noted, “We would have been inclined to agree with Felix more often in conference if he quoted Holmes less frequently to us.” Frankfurter confided in a letter that he knew some of his fellow justices “get sick and tired of hearing about Holmes and his genius … but it’s a state of mind I can’t understand. I belong to the Ecclesiastes school. ‘Let us praise famous men.’ ” Frankfurter simply could not stop himself from engaging in Holmes idolatry. Indeed, it seems difficult to escape the conclusion that Felix Frankfurter, the crown prince of judicial restraint, possessed far too little self-restraint.

In the end, Frankfurter was unwilling to dedicate the time required to produce vital, enduring written opinions, the most significant measure of any justice. One journalist’s early assessment of Frankfurter’s writings was telling: “Press excitement over the first opinions handed down by Justice Frankfurter cooled noticeably when the reporters began to read them. They were tough going.” The going did not get much easier over time, as Frankfurter made preciously few distinctive contributions to the canon of American constitutional law. Frankfurter’s opinions, even at their best, sound like nothing so much as an Oliver Wendell Holmes cover band.

In one of Snyder’s more revealing asides, he notes that Frankfurter organized his chambers in an idiosyncratic fashion. Unlike his fellow justices, he did not claim for himself the grandest room in the office suite accorded each member of the Court—the one featuring a fireplace, a bathroom, and, most important, some solitude. Instead, Frankfurter, his law clerks, and his secretary all worked together cheek by jowl in a center office. One former law clerk noted that the justice

was interested in everything. By eight in the morning, he had read five newspapers. He’d already discussed foreign affairs … and taken a stroll with Dean Acheson. By the time we law clerks arrived at the office at nine, he’d be ready to give us a seminar on government until ten or eleven.

The office arrangement suggested, though, that Frankfurter was patently uninterested in at least one thing: devoting long, lonely hours to crafting first-rate opinions that would shape the law for future generations. He preferred to admire the tenor of his own voice. But casual talk, even when performed by a virtuoso, is ultimately ephemeral.

This article appears in the September 2022 print edition with the headline “‘The Greatest Talker of His Time.’”