It’s Bill Murray: A Facial Recognition Algorithm Settles the Debate

Actually, though, maybe it’s Tom Hanks.

Bill Murray smiles for a camera after arriving at the Kennedy Center for the Performing Arts for the 19th Annual Mark Twain Prize for American Humor presented to Bill Murray on Sunday, Oct. 23, 2016, in Washington.
Bill Murray, photographed at the Kennedy Center for the Performing Arts, in October 2016. (Owen Sweeney / Invision / AP)

A friend and I had the same reaction the first time we saw the photo: How is this even a debate?

It’s so obvious, we agreed.

Then, simultaneously: “It’s Tom Hanks.” “It’s Bill Murray.”

Such is the nature of this week’s internet mystery, a puzzle that’s as inconsequential as it is intriguing. It begins with a photo of a crying baby uploaded to the Facebook group “Reasons My Son Is Crying” back in 2013. The reason for the tot’s tears, as explained in the caption: “He met Bill Murray.” The famous actor standing with the child is mock wailing, matching the boy’s expression:

Except, wait a minute, was that really Bill Murray?

“Totally thought that was Tom Hanks...” one commenter wrote.

“Totally thought it was Tom Hanks too,” another added.

“Tom Hanks!” someone else chimed in.

Let’s take a closer look:

The woman who had shared the image in the first place attempted to settle the debate: “It’s totally Bill. I can vouch cause I'm the one in the photo with the massive grin thinking ‘oh my god this is going to be an awesome photo!’”

This didn’t satisfy everyone. The debate—which began when the photo was first posted in 2013 and was revived this week—continues from there.

In making their case for either Hanks or Murray, people examined eyebrow bushiness and the contours of the actor’s left ear. They looked for telltale scars and forehead crinkles. “What I learned from the comments: When Bill Murray is crying, he starts to look like Tom Hanks,” one person wrote.

One of the weird aspects of all of this is that Tom Hanks and Bill Murray are both ultra famous and don’t normally look very much alike. Computer scientists have even studied Tom Hanks’s face to reverse engineer what it is about Hanks that makes him look like himself.

Last year, the computer scientist Ira Kemelmacher-Shlizerman and her colleagues trained a machine to comb through a huge photo database of human faces, which included Hanks, and had an algorithm map dozens of points on each face to track changing facial expressions. Then, they used what the computer had learned to create uncanny 3-D simulations of celebrity likenesses. There are many implications for this kind of technology in augmented reality, virtual reality, and filmmaking—all areas where you might want to simulate the appearance of a certain individual’s face.

These kinds of face simulations also pose the potential for more confusion down the road. One of the ways computer scientists test the accuracy of their work is to map one person’s distinctive facial expressions onto a simulation of another famous face. The idea is that if their model is good enough, Hillary Clinton still looks like Hillary Clinton, even when she’s making a Tom Hanks expression. The images below give you a sense of how this works, with the top row showing the original (real) expressions being mimicked, and the subsequent rows showing simulations of what other famous faces would look like making those same expressions. (And if you think this stuff is creepy, just wait until you see the related video simulations.)

Kemelmacher-Shlizerman focused on Hanks in her work because he comes with a trove of data—simply because he’s been so famous for so long, there are tons of photographs of him out there. And computers need lots of training photos to reliably recognize faces and other patterns.

When I shared the mystery photo with Kemelmacher-Shlizerman this week, she saw Hanks: “This looks like Tom Hanks based on all the work we did in 3D,” she told me, “but can’t say definite.”

What would a machine say?

To find out, I contacted Pinar Yanardag, a computer engineer at MIT who specializes in machine learning.

“A machine-learning algorithm can definitely be trained on images of Bill Murray and Tom Hanks and then predict which actor is the most similar to the image,” she said. But such an experiment would be relatively time consuming to set up. A shortcut, she suggested, would be to consult one of the many facial-recognition algorithms we already have at our fingertips.

“Even though we don’t have a machine-learning algorithm in hand, in a way, this is what Google Image Search is essentially doing,” Yanardag said. “It is trained on millions of images, and based on the image you’ve uploaded, it finds the most similar description to it. So it will essentially do the same job with what could be done via a custom one we develop.”

This isn’t a foolproof method: Upload the original image in question, for instance, and many of the search results lead you back to the debate over who is in the photo. After all, Google’s algorithm is relying on descriptions provided by users, and many of those descriptions include text like, “Is this Tom Hanks or Bill Murray?!”

There’s a way around this, though. Yanardag cut out a tiny square of the photo—just enough to include the actor’s eyes and nose—and uploaded that excerpted image to Google.

“In this case, it reveals the picture as Bill Murray,” she wrote in an email. “Quite surprising to me since I thought he was Tom Hanks.”

Google’s algorithm says this is Bill Murray.

Of course, this method has its own problems. (“A machine will likely make mistakes on this one,” Kemelmacher-Shlizerman had warned me.) I uploaded just the mouth from the photo to see how Google’s algorithm would contextualize it and was told that the image was probably a picture of a newborn baby.

But, hey! That’s pretty good, actually.

Or, at least, it’s exactly the face Bill Murray (Tom Hanks?) was trying to make in the first place.