What's It Like to See Ideas as Shapes?

Thoughts and feelings are constellations in the mind of a man with a rare form of synesthesia.

Jonathan Jackson's synesthesia often makes his thoughts appear as points moving in unison, analogous to birds flying across the sky. (Amir Cohen / Reuters)

One spring evening in the mid 2000s, Jonathan Jackson and Andy Linscott sat on some seaside rocks near their college campus, smoking the kind of cigarettes reserved for heartbreak. Linscott was, by his own admission, “emotionally spewing” over a girl, and Jackson was consoling him.

Jackson had always been a particularly good listener. But in the middle of their talk, he did something Linscott found deeply odd.

“He got up and jumped over to this much higher rock,” Linscott says. “He was like, ‘Andy, I’m listening, I just want to get a different angle. I want to see what you’re saying and the shape of your words from a different perspective.’ I was baffled.”

For Jackson, moving physically to think differently about an idea seemed totally natural. “People say, ‘Okay, we need to think about this from a new angle’ all the time!” he says. “But for me that’s literal.”

Jackson has synesthesia, a neurological phenomenon that has long been defined as the co-activation of two or more conventionally unrelated senses. Some synesthetes see music (known as auditory-visual synesthesia) or read letters and numbers in specific hues (grapheme-color synesthesia). But recent research has complicated that definition, exploring where in the sensory process those overlaps start and opening up the term to include types of synesthesia in which senses interact in a much more complex manner.

Jackson sees his thoughts as shapes. Every person he meets, every sentence he reads, and every decision he makes are presented as data points on a kind of continuously moving mental scatter plot, creating figures he compares to constellations. If he were to make a decision about whether to take a new job, for instance, those points might represent salary, location, and cost of living. The lines between them would change depending on how attractive they were to Jackson, creating a unique configuration for each option.

For many people, decision-making is a murky, difficult process. Think it through, go with your gut, follow your heart—there’s a reason the English language features so many ways to talk about groping around in the cellar of the conscience to find the light switch of intuition. But for Jackson, intuition is anything but blind. When he makes a choice, his gut feelings are visually laid out in front of him. He can choose among his options the way others might choose the reddest, glossiest apple from a bowl.

In 2014, Jackson, who now lives in Seattle, designed and released an app with the aim of helping others make decisions. Called ChoiceMap, it helps users weigh the factors in a given decision, using an algorithm to spit out a “perfection score” for each option. Cold hard numbers; no more groping in the darkness. The app, which was rebuilt and re-released in May, helps users view decisions through a mathematical prism. It also serves as a reminder of the distance between people’s perceptions that can never be fully crossed—and as a testament to just how delicate and strange communication is in the first place.

* * *

It’s possible that someone could know Jackson for years without realizing anything is different about him. He’s clean cut, with an open face that masks a steely, almost stubborn resolve. He plays guitar and goes for long walks in his free time.

Sometimes, though, as we’re talking, I see his eyes flit to the side or focus elsewhere. That’s when he says he’s trying to “translate” his thoughts, from complex mathematics into something he might be able to convey in words. Early on, scheduling an interview, he explains that calls with strangers can be difficult. Without prior social context, his thoughts about a new person sometimes blur—he compares the experience to a nebula instead of a constellation—and that makes it hard to concentrate. So, we Skype instead.

Little disconnects of this kind started early for Jackson. Just as he did in his late-night conversation with Linscott, he remembers climbing on furniture to “look” at the ideas in his schoolwork from a different angle as a kid. Since nobody else around him did that, he learned to hide it. “For more than two decades, I didn't know why my brain worked differently than others,” he says.

A native Californian, Jackson went east to Massachusetts for college—that’s where he met Linscott. From there, he moved to Princeton to study the cognitive science of religion, examining the ways the human mind acquires and shares religious belief. “I wanted to understand how other people thought,” he says. “It felt like I had spent my life translating in one direction.” This was before his diagnosis, when he was aware of a yawning gap between his thoughts and other people’s but could not explain it.

Trying to bridge that gap could be frustrating. Jackson seemed to do well at Princeton, but underneath a gregarious façade, he was struggling. At his lowest point, he was spending up to twenty hours a day in the library poring over cognitive science research, in search of an explanation for what made his, and all brains, work. “By then it had become difficult to interact,” he says. “It exhausted a lot of people.”

Sometimes, walking home after a long study session, Jackson’s visual field was so cluttered with ideas that he had to sit down, call his girlfriend, and ask for a ride.

“The best metaphor I can give: When you were a kid did you ever just stare at a flashlight and turn out the light in the dark, and the light seems like it’s there flashing in front of your eyes?” he says. “That’s what it’s like, only with high-dimension geometry.”

* * *

Jackson’s difficulties reflect a larger issue in synesthesia research: the difficulty of translating, or even attempting to explain, individual perception. Trying to understand how someone else sees the world can be like picturing an animal you’ve never seen based only on that person’s description. And that, unsurprisingly, can lead to misunderstanding.

For researchers, tackling this requires asking questions about biology and the building blocks of perceptual experience. Does synesthesia stem from something as fundamental as a sound wave hitting an eardrum, for instance? Or does it arise further along the thought assembly line, where our understanding of sensations is processed and expressed? This distinction can be traced to a foundational 1929 study that asked subjects to name and ascribe traits to two shapes—one spiky and jagged like an uneven star, the other bulbous and blobby like a mutated flower. The study, and a follow-up performed in 2001, found that, given two name options, speakers across languages and cultures chose “baluba” in the earlier study and “bouba” in the later study for the curvy blob, and “takete” in the earlier study and “kiki” in the later study for the spiky star. And not only that, they also tended to agree on which shape would be nervous, which laidback; which high-class, which low-class; and a number of other traits.

Today, nearly 90 years later, Danko Nikolic, a researcher at the Max Planck Institute in Germany, draws from “bouba and kiki” to argue that it is the ideas synesthetes use to categorize sensory information that prompt synesthesetic experiences. Our ideas and our senses are inextricably linked, he says, and that means (at least in the case of synesthesia) that one can’t exist without the other. He points to one study, in which synesthetes who see numbers or letters in specific colors were shown a shape that could be interpreted as an “S” or a “5” in a field of other symbols. When those symbols were numbers, most of them saw the shape as a “5”; when the symbols were letters, that interpretation changed to an “S”—and the corresponding color changed with it.

Other scientists see synesthesia as a crossing of wires at the most basic level. These scientists contend that synesthetes start processing sensory information differently from the very first moment of contact—for instance, before the brain can even name wavelengths of light that bounce into the eye as “blue.” Yet most researchers argue for a more ambiguous middle ground, in which both perception of a figure’s form and the act of naming it work together to create synesthesia. Yes, synesthetes respond to visual stimuli in fundamentally different ways than other people, they say. But because the brain is a natural predictor, it begins suggesting categories for those wavelengths—names like “blue”—at the very beginning of sensory processing. Perhaps the way ideas and perception interact can’t be so neatly defined, after all.

* * *

That middle ground theory is also where science begins to explain—and test for—more complicated synesthesia like Jackson’s. With simple synesthesia, such as grapheme-color, testing easily identifies synesthetes. One of the most popular assessments asks a subject to match the shade he or she sees in a letter or number using a computerized color palette. “You come back as a surprise a month or six months later and say, ‘Find your color again,’ and they pick the same one,” Nikolic explains. A non-synesthete is hard pressed to do that.

For someone with more complicated synesthesia, testing is trickier. Nikolic has worked with people who see time as a line, a spiral, or more complicated geometric shapes. The key to testing them, he says, is asking for “really, really fine details. You wait a long time, you surprise them, and then do it again.”

Still, in discussing something as personal as perception, testing can only get you so far. Researchers focusing on more complex synesthesia must rely on subjective descriptions and observation from their subjects. In that vein, Jackson sends me to the website of Venezuelan artist Rafael Araujo, whose elaborate helix and nautilus drawings he compares to his own thought processes. People in Jackson’s circle report a pattern of behavior consistent with the thinking he describes, including use of visually-oriented language, frequent discussion of ideas in terms of shape, and offering to “rewind” a conversation or reconstruct it in an explicitly visual way.

“I can tell when he’s paying close attention to his thought process and not paying as much attention to translating,” his wife, Rebecca Jackson, says. “It’s the way his eyes focus. Sometimes there’s hand movement.”

That’s something I also see during our interactions. Once, during a Skype call, I notice Jackson’s eyes moving to the side particularly frequently. Finally, we interrupt our conversation; a stray thought has him distracted, and it’s keeping him from the topic at hand. It’s as though his thoughts are points moving in unison, a flock of abstract birds, and this distracting topic is an intrusive wedge flying across the plane. The thought-birds scatter in its wake.

* * *

As persuasive as the evidence supporting his synesthestic experiences may be, the truth, of course, is that no one else can see through Jackson’s eyes. “I’ve never had anyone say ‘No I don’t believe you,’” he says. Still, when I ask whether people ever express disbelief at his abilities, he says, “There aren’t a lot of moments of me saying, ‘Hey, person I don’t know very well, let me share this unique way my brain works and then have you be incredulous about it.’”

It took time for him to achieve that equanimity. He climbed out of his graduate school depression only with the help of Dick Fenn, a Princeton Theological Seminary professor. Fenn remembers a day in one of his seminars when Jackson, attempting to convey a complicated idea, drew some of his thoughts on the chalkboard. Instead of greeting Jackson’s drawings with scorn or confusion, Fenn wanted to know more. He invited Jackson for coffee after class.

That connection with Fenn marked the beginning of a new chapter for Jackson. “It flipped this switch for me,” Jackson says. Now, the conversation was on his terms. He was deeply moved by Fenn’s attempts to engage in a way no one had before; Fenn even encouraged him to “write” his master’s thesis in the style of his seminar drawing. Fenn was reaching out; he was trying to bridge the impossible gap.

At one point in our conversations, Jackson sends me a photo of his thesis, slightly blurry, but showing clearly the project’s sheer size and complexity. The drawing fills three successive chalkboards with minute writing, numbers, symbols. It’s organized around a kind of X-shaped axis, with bubbles and squares branching off, squiggly sections of ideas, tangents, roots, remarkable for what it is: human thought, cross-sectioned.

* * *

Jackson stresses that his synesthesia and ChoiceMap, his app, are independent—that he conceived of the idea separately and the app’s algorithm is not based on his own decision-making process. He often notes that his team is only 25 percent synesthete, emphasizing that the app represents his colleagues’ collective work, not just his. “ChoiceMap uses math,” he says. “Anyone can learn from and critique math. That’s the point; math is translatable.”

Still, it’s hard not to see parallels. When making decisions with the app, ChoiceMap users are prompted to list the factors they are considering, weigh those factors on a scale of importance, then evaluate their options on the basis of each factor. The app breaks everything down using an algorithm and offers its perfection score for each possibility its users could choose. Few people might think to use high-level math in their decision-making; for some, even a pro-con list is a stretch. It may well take someone who sees data points in his thoughts to make that connection.

Since its release in 2014, thousands of users have tasked ChoiceMap with helping them decide where to live, what car to buy, or whether to break up with their partners. It’s impossible to know whether they went through with those choices, but last summer Jackson did meet a baby who owed its existence to a ChoiceMap decision.

The newest version of ChoiceMap, released in May, includes many more decision “decks” available to the public, arrays of pre-set factors to help users choose between types of birth control or car models. Additionally, an add-on called ChoiceMap Medical is now in testing at the University of Pittsburgh Medical Center’s oncology department, inspired by emails Jackson and his team received from doctors using the app to make difficult decisions about their patients’ treatments. The new app helps oncologists and their patients consider factors such as memory loss, nausea, cost, and length of life when choosing cancer treatments.

Although the ChoiceMap Medical program is currently limited to patients with brain metastases, the team is working with multiple hospitals to expand coverage into general oncology, geriatrics, surgery, and beyond. “We’re switching to a healthcare focus,” Jackson says. “It’s the most meaningful use case any of us can think of. Helping people choose which car to buy is fine, but there are harder decisions.”

* * *

Any app is a literal brainchild of its designers; even Angry Birds is, in its way, a look into the mind of another human being. In that context, Jackson’s point about ChoiceMap’s use of math as translation is powerful. At root, every human interaction involves a similar transaction, however small: a shout across the chasm between minds, an imaginative jaunt into the great unknown outside our own brains. Following ChoiceMap’s decision-making protocols—teasing out individual factors from the cloud of fuzzy values and hunches, putting a number to their importance—is one uniquely concrete way to see the world from another angle, to bridge the chasm.

Jackson has learned to love that space between. His synesthesia still frustrates him occasionally—“You don’t get to switch that off on Friday night when you’re sitting around shooting the shit,” he says—but with help from Fenn and a close-knit support network, he has embraced it as an asset.

Linscott not only sees how far Jackson has come since they were intense kids smoking cigarettes; he suspects the compassion he so values in Jackson is connected to his friend’s synesthesia. “I can imagine how that would contribute to his ability to have deep conversation and to listen empathically,” he says, “because it’s not just words he’s processing. It’s something he’s seeing, something he can engage with on multiple levels.”

Despite all the times she’s seen Jackson blinded by his thoughts or frustrated by difficult translations, Rebecca, his wife, feels similarly. She believes synesthesia contributes something essential to her husband. It means he can be more creative, more thoughtful, more decisive. He can interact with his ideas in a way that is impossible for others.

She knows it’s not her decision, but she can’t help but wish: “If we’re able to have a child, I hope our child has it,” she says.