This App Reads Your Emotions on Your Face

Face-tracking technology allows computers to read facial expressions. In case you want to know about your mood.
More

If you've ever been confused about how you're feeling, and it happened to be the 1970s, you could always count on the mood ring. The jewelry fad claimed to read wearers' levels of anxiety or ebullience by measuring body temperature.

Today there's a more reliable—but equally far-out—app that performs a similar function: the clmtrackr, a new emotion-analysis tool created by a Norwegian computer scientist named Audun Øygard.

You turn on your webcam, stare into your screen, and the program will tell you what emotions you're experiencing, and in what proportions, from anger to sadness to joy. 

The facial tracking is accomplished through a technique known as the constrained local model, or CLM, a type of algorithm that draws on thousands of existing pictures of faces to identify facial features and predict how they'll look when the face is scrunched into a smile, for example, or drooping in a frown. 

"It has learned from prior training each of the facial landmarks," Jeffrey Cohn, a professor of psychology and robotics at Carnegie Mellon University, told me. "Then for a new face, it goes looking for those points that it has learned to find."

The green lines you see are made up of 70 different points, which track everything from the corners of the mouth, as they curve up, to the eyebrows, as they raise.

Clmtrackr is far from the first program to analyze facial expressions based on video input, but it's unique because it's written in JavaScript so it can run in a browser—there are no software downloads required.

The technology underpinning clmtrackr and similar programs is already being used by doctors to diagnose and treat patients with conditions like autism or depression.

"We've found meaningful changes in facial expression as people recover [from depression]," Cohn said. "When people are severely depressed, they tend to make many expressions of contempt or disgust that signal to other people, 'stay away, leave me alone,' but as they become less depressed, you see less of that and more smiles. You also see more sadness: The person who is feeling sadness wants more help from others. As someone is becoming less depressed, they might also be more sad."

In the business world, Cohn said advertisers use tools like these to determine the effectiveness of commercials based on viewers' reactions.

For everyday consumers, the artists Lauren McCarthy and Kyle McDonald created US+, a video chat application that employs face tracking and tonal analysis to help people appear more positive or less hostile in video chats (and possibly their corollary, real-life chats.) If you knit your brows and drone on about your partner's annoying tics, the app will show your "aggression" rating shooting through the roof.

"Clmtrack is a javascript implementation of a face tracking algorithm, so it's not very fast or precise yet, but with some more work it might be usable for some of these use-cases," Øygard told me in an email. "There is quite a lot of development in this field currently, so every few months new methods that are faster and more precise are published, and I think we'll soon see face tracking pop up in a lot more places."

Today, though, I used the clmtrackr just for lolz.

Other than surprise, the program doesn't work very well with glasses, so I removed mine when I was going through the other feelings.

For example, I was sad that the other day I dropped and broke my iPad:

Nevertheless, I was happy that I survived the Polar Vortex:

Then I tried it without making a face—that is, looking at the computer as I normally do—and it told me that I just looked a little bit sad:

Further proof that I, like many other people of my ethnic extraction, suffer from bitchy resting face.

Øygard also created a face substitution program that can swap anyone's visage for that of a celebrity's through a type of digital mesh that matches the contours of the face. Here's me as Kim Kardashian:

Fellas?

Jump to comments
Presented by

Olga Khazan is a staff writer at The Atlantic, where she covers health.

Get Today's Top Stories in Your Inbox (preview)

Social Security: The Greatest Government Policy of All Time?

It's the most effective anti-poverty program in U.S. history. So why do some people hate it?


Elsewhere on the web

Join the Discussion

After you comment, click Post. If you’re not already logged in you will be asked to log in or register. blog comments powered by Disqus

Video

Adventures in Legal Weed

Colorado is now well into its first year as the first state to legalize recreational marijuana. How's it going? James Hamblin visits Aspen.

Video

What Makes a Story Great?

The storytellers behind House of CardsandThis American Life reflect on the creative process.

Video

Tracing Sriracha's Origin to Thailand

Ever wonder how the wildly popular hot sauce got its name? It all started in Si Racha.

Video

Where Confiscated Wildlife Ends Up

A government facility outside of Denver houses more than a million products of the illegal wildlife trade, from tigers and bears to bald eagles.

Video

Is Wine Healthy?

James Hamblin prepares to impress his date with knowledge about the health benefits of wine.

Video

The World's Largest Balloon Festival

Nine days, more than 700 balloons, and a whole lot of hot air

Writers

Up
Down

More in Technology

Just In