How to Hear Sign Language

Microsoft uses Kinect technology to turn gestures into text, and text into speech. 
More

The most significant barrier between the deaf and the hearing is, generally, language. Signs and speech use different methods to express themselves; that divide alone makes the everyday translation of sign-to-speech, and speech-to-sign, particularly challenging. 

Scientists at Microsoft and the Chinese Academy of Sciences, however, think they have found a way to bridge the gulf. And it involves the same technology used in video games. The Kinect Sign Language Translator project, released this week in a prototype form, aims to enable the hearing to understand sign language—and vice versa. 

It works, essentially, like this: The deaf person signs, and the system renders both a written and a spoken translation of those gestures. (It uses Microsoft's Kinect to process the gestures of the signs.) The system also processes a speaking person’s words, converting them into readable text. 

Which means that the interactions between the deaf and the hearing could soon become much less friction-filled than they've been in the past. A deaf doctor could communicate more fluidly, and more meaningfully, with a hearing patient. A hearing store manager could communicate with a deaf patron. The pair, in their interaction, wouldn't need extra knowledge of each others' languages; they would just need a tool to do the translating for them. 

In the video above, Dandan Yin, a 22-year-old computer science student who was born deaf, demonstrates the translation system. You can see her gesturing to a Kinect device that is, in turn, connected to a sign language prototype. You can see words appearing on the screen, translating the signs.

There is more work to be done with all this: The system isn't instant, meaning that one source of social friction—awkwardly waiting for a translation—is still part of its process. And there are more words for it to learn. But the idea is there, and the framework is there. Any system based on machine learning will get, theoretically and almost inevitably, better with age. You can see how, with incremental improvements, this is a technology that can bridge the gulf between the deaf and the hearing. As Stewart Tansley, director of Natural User Interface for Microsoft Research Connections, puts it in the video: "What, overwhelmingly, you feel when you see it working is a certain magic."

Jump to comments
Presented by

Megan Garber is a staff writer at The Atlantic. She was formerly an assistant editor at the Nieman Journalism Lab, where she wrote about innovations in the media.

Get Today's Top Stories in Your Inbox (preview)

What Is the Greatest Story Ever Told?

A panel of storytellers share their favorite tales, from the Bible to Charlotte's Web.


Elsewhere on the web

Join the Discussion

After you comment, click Post. If you’re not already logged in you will be asked to log in or register. blog comments powered by Disqus

Video

The Death of Film

You'll never hear the whirring sound of a projector again.

Video

How to Hunt With Poison Darts

A Borneo hunter explains one of his tribe's oldest customs: the art of the blowpipe

Video

A Delightful, Pixar-Inspired Cartoon

An action figure and his reluctant sidekick trek across a kitchen in search of treasure.

Video

I Am an Undocumented Immigrant

"I look like a typical young American."

Video

Why Did I Study Physics?

Using hand-drawn cartoons to explain an academic passion

Writers

Up
Down

More in Technology

Just In