What It Means That Computers Can Tell These Smiles Apart, But You Can't

Computer algorithms can now discern the meaning behind humans' facial expressions. 

[optional image description]
Image via Mohammed Hoque, Daniel McDuff, and Rosalind Picard.

The four people above are taking part in a clinical experiment. In the screenshots shown here, each guy is smiling once out of delight (reacting to a picture of an adorable baby) and once out of frustration: made to complete an online form that keeps malfunctioning. Awww vs. argh: same general expression, totally different emotion. 

So which is which? Who's smiling out of joy, who out of annoyance? 

If you're not totally sure, you're not alone. We humans need context and narrative to be able to discern the meanings of our fellow humans' facial expressions. We're sensitive to subtleties. That's one thing that makes us different from machines. 

Except ... when it's not. In a paper just published in IEEE Transactions on Affective Computing, Mohammed Hoque, Daniel McDuff, and Rosalind Picard share a system that allows computers to become as sensitive as -- and, in fact, even more sensitive than -- humans. 

The team, members of MIT's Affective Computing Group, combined two insights to arrive at their algorithm. First, genuine smiles tend to build slowly and linger, while frustrated smiles tend to appear and disappear quickly. Second, the musculature of fake smiles tends to differ from that of genuine ones: hence "thin" smiles, "stiff" smiles, etc. 

Those types of smiles are often involuntary. When Hoque and his colleagues asked study participants to feign frustration, 90 percent of them did so without smiling. But when the researchers presented their subjects with a task that caused genuine frustration -- filling out an online form, only to find their information deleted after they pressed the "submit" button -- 90 percent of them ended up smiling. Frustratedly.

The algorithm Hoque and his colleagues developed accounts for that expressive difference. And it does so quite effectively. The team's computer-based system was able to figure out which smiles were fake 92 percent of the time. The success rate for humans who were asked to do the same: 50 percent, which is obviously the same as if they had randomly guessed.

The most immediate and obvious application of the team's findings would be to aid people diagnosed with Autism Spectrum Disorder. Emotion-reading computer programs could help the autistic to assess and interpret other people's facial expressions -- one of the biggest impediments to social interaction.

But what about the broader implications? First, a hope ... then, a caveat. 

On the one hand, as the paper puts it, the team's findings could be used "to develop automated systems that recognize spontaneous expressions with accuracy higher than the human counterpart." Facial recognition is now a fairly common technology, used in everything from Facebook to city streets. Emotion recognition is the next logical step in that progression -- a field that could bring a whole new meaning to "sentiment analysis."

Presented by

Megan Garber is a staff writer at The Atlantic.

How to Cook Spaghetti Squash (and Why)

Cooking for yourself is one of the surest ways to eat well. Bestselling author Mark Bittman teaches James Hamblin the recipe that everyone is Googling.

Join the Discussion

After you comment, click Post. If you’re not already logged in you will be asked to log in or register.

blog comments powered by Disqus


How to Cook Spaghetti Squash (and Why)

Cooking for yourself is one of the surest ways to eat well.


Before Tinder, a Tree

Looking for your soulmate? Write a letter to the "Bridegroom's Oak" in Germany.


The Health Benefits of Going Outside

People spend too much time indoors. One solution: ecotherapy.


Where High Tech Meets the 1950s

Why did Green Bank, West Virginia, ban wireless signals? For science.


Yes, Quidditch Is Real

How J.K. Rowling's magical sport spread from Hogwarts to college campuses


Would You Live in a Treehouse?

A treehouse can be an ideal office space, vacation rental, and way of reconnecting with your youth.

More in Technology

Just In