u_topn picture
rub_dc picture
Atlantic Unbound Sidebar

Previously in Digital Culture:

"Where the Rubber Meets the Road," by Ralph Lombreglia (March 1998)
A brand-new graduate program at a tiny Vermont college is founded on a simple yet far-sighted idea: the next phase of the digital revolution depends as much on education as on technology.

"The World Accord to David Gelernter," by Harvey Blume (January 1998)
An interview with a computer scientist who argues that beautiful technology -- and a return to traditional values -- must show us the way forward. Plus, a series of excerpts from David Gelernter's Machine Beauty: Elegance and the Heart of Technology.

For more, see the complete Digital Culture Index

Discuss this topic in the Digital Culture forum of Post & Riposte.

A Function Specific to Joy
Are we ready for computers that know how we feel?

by Harvey Blume

April 29, 1998

It is no longer accurate to think of emotion as some kind of evolutionary relic or as some kind of luxury, unnecessary in either case for intelligence. Instead, emotion has a critical role in cognition and in human-computer interaction.
--Rosalind Picard

Affect, until now, has served as a fairly reliable borderline between human and machine: humans have and recognize emotions, computers don't. Affective computing is a new focus within computer science, devoted to creating computer systems that emulate and recognize human emotions. It's a field that opens the way to previously inconceivable kinds of computer applications at the same time as it threatens to disturb the time-honored language we've developed for addressing and expressing our emotions. Opening this border between human and machine can't help but unsettle things even as it creates conveniences.

Rosalind Picard
Rosalind Picard

MIT's Rosalind Picard, a pioneer of affective computing (her full titles are NEC Development Professor of Computers, and Communications Associate Professor of Media Technology), believes that the affective barrier between people and machines has long since outlived its usefulness. As she writes in her book, Affective Computing (1997), "Computer-based communication is affect-blind, affect-deaf, and generally speaking, affect-impaired. A quantum leap in communication will occur when computers become able to at least recognize and express affect."

Despite Picard's intimations of the potential importance of affect to the human-machine interface, she did not exactly rush into work on affective computing. "I'm used to being one woman in a room with one hundred men studying engineering," she told me recently. "And we always downplayed emotions. Emotions are seen as a negative thing, sometimes associated with the emotional woman." But breakthroughs in neuroscience meant that Picard could no longer pretend that reason alone was a legitimate subject of study, that emotion was somehow disreputable.

New, increasingly precise brain scanning techniques, among them magnetic resonance imaging (MRI), have encouraged a broad group of thinkers -- Daniel Dennett in philosophy and Steven Pinker in evolutionary psychology, to name a couple -- to believe that a codex to the human brain may at last be within reach. If you can see what goes on within the skull when people read, think, emote, dream -- if you actually can correlate neural pathways to mental activity -- all previous attempts to understand the mind seem superficial. Brain scans taught Rosalind Picard that even when the brain is involved in learning, the limbic system, the seat of the emotions, will sometimes light up and become active while the cortex, the seat of reason, shuts down. The situation, then, so far as reason and emotion are concerned, is obviously a lot more complicated than computer science had allowed. You can no longer choose to retain reason and discard emotion, not if you want to build computers that perceive and make decisions along the lines that human beings do.

For a description of the full panoply of current brain scanning techniques, see The Engine of Reason, the Seat of the Soul: A Philosophical Journey into the Brain (MIT Press, 1995), pp. 278, 300.

Affective Computing When neuroscience convinced Picard that "if you disconnect emotions from the reasoning part of the brain, the reasoning part of the brain doesn't work properly," her transition to affective computing became inescapable. Along with it came a vision of a new array of computer applications, from the functional to the nearly decadent. Emotion sensors, for example, are a key element in affective computing's prospective arsenal and might, when they come online, enable voice synthesizers, like the one Stephen Hawking uses, to detect their owners' feelings and inflect their utterances accordingly. Emotion sensors might also, by recording signs of frustration and annoyance in users, tell software makers where unforeseen problems with their new programs lie. And, in a world made slightly more perfect by their presence, emotion sensors might tell when you've entered a flow state in your work, and turn off all electronic interruptions for the duration.

Mood detection will be a distinct sub-specialty of affective computing. Orpheus, an Affective CD Player being researched by Picard's Affective Computing Group at MIT's Media Lab, will recognize your mood, and then will spin up mood music to go along with it. Picard is also working on mood-based data retrieval -- a system that would allow you to select photographs, for example, not by the objects represented but by the mood evoked. On the most intimate level, affective computing heralds the age of wearables, small computers that double as jewelry, eyeglass attachments, or clothing accessories. Picard sees people and their wearables as entering into a symbiotic relationship: research is underway, according to Picard, "to make wearables human-powered, at least in part, by using the energy people expend in natural activity."

Related link:

"Are emotions a desirable property for computers to have? It's hard to imagine someday walking into a computer store and saying, 'Give me the most emotional machine you've got.' After all, isn't possessing the highest form of rationality one of the hallmarks of computers?"
--Rosalind Picard, from "Does HAL Cry Digital Tears? Emotion and Computers," in HAL's Legacy: 2001's Computer as Dream and Reality, edited by David Stork (MIT Press, 1996).

Wearables will play a role in health care, telling you which activities cause stress, and in your social life, linking you to a mood network and advising you about whom, in your current state of mind, you might want to connect with. As wearables pick up an individual's literal and metaphorical smell, they will, according to Picard, "like underwear, probably cease to be shared and will become truly personal computers."

Some of this, to be sure, is futuristic in the extreme, as is Picard's suggestion that "computers might recognize emotions and other states that humans would not ordinarily recognize." But before we reject the notion of computer hypersensitivity to emotion, it might be useful to remember that computers can make sounds that elude other instruments -- micro, sampled, or combinatory tones. The problem of exhibiting feelings that combine or are arbitrarily (and perhaps inhumanly) situated between other feelings may not be so different, from the computer's standpoint, from the problem of playing micro-tones or mixing the timbre of a trumpet, say, with that of a violin. Today's computers can generate plausible images of how our faces will look as they age. Tomorrow's may routinely provide print-outs of how anger looks when layered over fear on a human face, or how anxiety looks when sprinkled on top of sexual desire. Picard again brushes up against a science-fiction world (that may one day be our own) when she writes of affective computers becoming too personalized, too saturated with "affective markers," to fix or throw out when they malfunction. The only way to make them normal again will be to subject them to "un-learning, to 'heal' the malfunctioning responses."

Of course, there's the question of how we get from where we are now to a situation in which a wearable might need therapy in order to recover from its owner's hot night on the town. My Mac flashes a smiley face during start-up -- and Picard considers that smile a first stab at affective computing -- but has no idea how I feel about its crashing just as I reach a Web site. How can it learn what a truly dim view I take of crashing? How, in general, can computers get a grip on our emotions?

Picard's answers can be disconcerting. A random walk through Affective Computing turns up many models of the emotions, each of which zeros in on our affect in a way only a computer could appreciate. Try on the following equation for example:
IF D(p, e, t) > 0

THEN set Pj(p, e, t) = fj(D(p, e, t), Ig(p, e, t))

where fj() is a function specific to joy.
Or consider the heat and motion maps of the human face a computer can use to assess whether we're feeling surprised, say, or totally disgusted. What's disturbing about this is not its implausibility, but precisely the opposite -- it can be done. The process is well underway. Inspired by Picard's work, IBM and British Telecom have already established their own affective-computing groups. Picard predicts that the first wearables will reach the market within the decade. Computers may eventually, just as she suggests, get to know us as well as or better than we know ourselves, gauging us by parameters we would never think consciously to employ, calculating our states of mind by speech rate, voice quality and intensity, measuring our current pulse and perspiration and comparing them to baseline figures. And we will learn to see ourselves, gradually, and at least in part, the way computers see us -- the way, for example, an alien might hone in on and track a human face.

It may be instructive to think about our emotions in terms of the mechanical models that affective computers will use to grasp them. For example, would you say that two conflicting emotions operate more along the lines of a microwave oven, which is limited to pure states, on and off, or more like a bathtub, where hot and cold mix and "mingle and form a solution that has a new state -- warm"? Picard concludes that both the microwave and the bathtub metaphor have their uses: melancholy, for example, may be the final product of a microwave-like alteration between love and sadness, whereas wariness is more likely brewed in an affective bathtub where interest comes pouring out of one spigot and fear out of the other.

And there is likely to be much good humor on the way to affective computing. Hugh Kenner -- in a strange, dense, and provocative book titled The Counterfeiters: An Historical Comedy (1968) -- speaks of computers as essentially parody machines that satirize their builders. (He speaks of Alan Turing as a master satirist, and discusses the Turing Test in the same breath as he does Jonathan Swift.) Well, I've seen Rosalind Picard give a demonstration of affective computing at MIT's Media Lab, and I am here to testify that when you download an affect -- in this case, confusion -- by means of Groucho Marx-like eyeglasses that measure furrowing in the telltale muscles between the eyes, and then display the results on a confusionometer, the process paves the way for a whole new kind of cyber-comedy. The Woody Allens and Steve Martins of the future will be attached to wearables that somehow, maddeningly, go wrong.

In the end, it seems to me that the advent of affective computing can't help but promote a process begun by Copernicus and furthered at every stage of the ongoing scientific revolution. It looks like the sun goes around the earth, but it doesn't. We think we know how to talk about our emotions, but even here, in the most intimate preserve of our humanity, it turns out we have just been guessing. With the heart as in the heavens, reality runs counter to intuition; to keep in step with the advance of science we turn to a combination of brain scanning and computer modeling to tell us what is really going on in the emotions. Of course, it's good to know how the emotions really work, what neural pathways light up when, and how computers can mimic those connections -- and it will definitely be useful to have computers around that are not emotional morons -- but the price will be a bit more alienation, a bit more digital distancing from ourselves. Digital culture has a way of seeping into our bones, altering our self-conception. Inevitably, it seems to me, we'll be persuaded by the example of affective computers to stop saying, "I'm sad," and to describe the emotion, if only ironically, by way, for example, of the Hidden Markov Model. After all, it's good enough for our wearable.

joydiagr picture
The Hidden Markov Model (from Affective Computing)

Rosalind Picard's view of her own work couldn't be more diametrically opposed to this skeptical, not to say jaundiced estimate. If I worry that by supplying digital recipes for our emotions affective computing will diminish us, Picard sees connecting to our "affective bits" as a way of honoring rather than mocking our humanity. As her personal Web pages attest, Picard is a practicing Christian, and her faith plays no small role in her involvement with affective computing.

"When you first start understanding how the brain works, it's amazing," she told me. "Even scientists who have no old-time religion will feel that sense of awe at how wonderfully we're made. Digging into the models of how the emotions work, I find I feel even greater awe and appreciation for the way we are made, and therefore for the Maker that has brought this about. My trying to understand all this is an act of worship, an act of homage, not an act of trying to usurp the Maker."

I think Picard is justified in believing that affective computing pays all due respects to our complexity. But I think a Hugh Kennerish suspicion that affective computing will propel us ever deeper into an age of irony is no less legitimate. Affective computing, like every major breakthrough in digitalization, will enlarge and diminish us, stimulate the imagination and impoverish it. At every step of the way we'll scrutinize the process, asking what has been gained, what has been lost. And it may be that our wearables, should we choose to put them on, will nudge us toward the answer.

Join the discussion in the Digital Culture forum of Post & Riposte.

More on Digital Culture in Atlantic Unbound

At a critical point in his life, Harvey Blume chose English over C and therefore writes reviews, criticism, and even the occasional book rather than computer programs. The co-author of
Ota Benga: The Pygmy in the Zoo (1992), he writes about art, literature, and new media.

Copyright © 1998 by The Atlantic Monthly Company. All rights reserved.
Cover Atlantic Unbound The Atlantic Monthly Post & Riposte Atlantic Store Search