Late in his life, Albert Einstein published the nearest he ever came to an autobiography. It’s scant on actual details of his life, on his triumphs and fiascos, loves and losses. Instead, Einstein wrote an account of what a life in science offers those who commit to it.

Like any good quest story, Einstein’s begins with sense of something missing. “Even when I was a fairly precocious young man,” he wrote, “the nothingness of the hopes and strivings which chase most men restlessly through life came to my consciousness with considerable vitality.” Against that void Einstein discovered that “out yonder there was this huge world, which exists independently of us human beings and which stands before us like a great, eternal riddle, at least partially accessible to our inspection and thinking.” From the time he was 12, he wrote, “The contemplation of this world beckoned like a liberation.”

I thought of that credo when I read about a pending Supreme Court case called Fisher v. University of Texas, where the court is set to decide whether the university’s affirmative-action policy unfairly promotes less-qualified minority students over white ones. During oral argument, Justice Scalia’s suggestion that a softer, kinder separate-but-equal system might be sufficient for African Americans drew most of the immediate backlash—as it should have. But it was Chief Justice Roberts who made the more subtle attack on the concept of diversity, when he asked “what unique perspective does a minority student bring to a physics class?” before musing, “I’m just wondering what the benefits of diversity are in that situation.”

Roberts seems to be suggesting that physics is a realm apart from say, the humanities, where unique cultural perspectives would be more obviously valuable—and in doing so, he gives voice to a widely held misconception about science. Roberts’s error is to treat physics as a discipline that sits outside its own history and the larger culture, when of course it does no such thing.

This was not the view of physics that 70-year-old Albert Einstein described as he looked back across his own life experience. Most of his not-really-an-autobiography limns the evolution of his thinking about his two theories of relativity. In describing the paradox that led him to his early breakthrough, the Special Theory of Relativity, Einstein noted that it took him a decade to get from his first thoughts to the final theory. (He started at 16, so he has some excuse.) The breakthrough turned on his realization that measurements of time and space aren’t absolute. Rather, they shift for different observers depending on how they’re moving relative to each other.

As he struggled to finish the theory, Einstein found that “even scholars of audacious spirit and fine instinct can be obstructed in the interpretation of facts by philosophical prejudices.” Einstein himself had to reach out of physics to develop the habits of mind that allowed him to see past the prejudices that obscured the relativistic universe he ultimately discovered. “The type of critical reasoning which was required for the discovery of this central point” he wrote, “was decisively furthered, in my case especially, by the reading of David Hume’s and Ernst Mach’s philosophical writings.”

David Hume! Benjamin Franklin’s friend (they corresponded on the matter of lightning rods, among other matters) and Adam Smith’s confidante! Hume’s name isn’t usually linked to deep physical insights. But as Einstein suggests, the intellectual work of physics doesn’t occur in a vacuum. It can’t be separated from a person’s learned habits of thought, from a particular set of lived incidents, books read, alternate worlds imagined. The diversity of Einstein’s particular background, sensibility, and cultural circumstances all played a role in bringing Special Relativity to fruition.

Roberts’s question about the “benefits” minorities might bring into a physics classroom suggests a classroom in which nothing outside physics may usefully impinge. That is, at best, a fatally narrow view. Roberts is thinking only about the answers, not the process of arriving at them. Actually doing science involves everything about the person doing the work—as, for example, the way Einstein turned his anger and pity for his father, a casualty of the rat race, into the goad that led him to so much of modern physics.

The case for diversity in American education (and life in general) turns on any number of reasons, from the historical to the instrumental (why throw away so much talent?). Einstein’s story suggests another: Physics, like any worthwhile inquiry, is not just a body of facts and methods. It is a way of being in the world that requires the full range of human experience.

A week after Roberts spoke from the Supreme Court bench, I heard the physicist Kaća Bradonjić tell a story at a Story Collider event in Cambridge, Massachusetts. Bradonjić told her assembled audience about her childhood in the then-Yugoslavian city of Sarajevo, and of the time after Yugoslavia collapsed, when she found herself living in the capital of Bosnia. When she was still just 11, Sarajevo became a battlefield. Her family escaped the savaged city just before the siege tightened sufficiently to bar any hope of rescue. In exile, Bradonjić became yet another person, a war refugee, her country, her language, her map of reality—all lost.

As her memory played out on stage, Bradonjić confronted the way time and space fragmented for her as a refugee. She told of trying to hold Sarajevo in memory as she grew up, the geography that took her from her home to the bakery, to relative’s houses, to school. She recalled the sequences of events before the shells started to fall, and the fracturing that followed when they did. Later, over years through which her city morphed in memory, she told us how she encountered the work of Albert Einstein.

The grown-up Bradonjić has become a general relativist, someone who uses Einstein’s grandest discovery to study how space warps and time swerves as events occur within the universe. The contemplation of our malleable cosmos has surely offered her some measure of liberation, a long road home to what it meant to have once lived in her beloved Sarajevo. Equally, inevitably, Bradonjić’s life frames her every attempt to understand the geometry of the universe.

Chief Justice Roberts’s view of the physics classroom is too narrow to admit this diversity of human experience. If such a view prevails, minority students will feel the blow first, but so will science, which is, after all, a search not just for knowledge, but for meaning.