Oh, that Ken Jennings, always quick with a quip. At the end of the three-day Jeopardy! tournament pitting him and fellow human Brad Rutter against the IBM supercomputer Watson, he had a good one. When it came time for Final Jeopardy, he and Rutter already knew that Watson had trounced the two of them, the best competitors that Jeopardy! had ever had. So, on his written response to a clue about Bram Stoker, the author of Dracula, Jennings wrote, "I, for one, welcome our new computer overlords."
Now, think about that sentence. What does it mean to you? If you are a fan of The Simpsons, you'll be able to identify it as a riff on a line from the 1994 episode, "Deep Space Homer," wherein clueless news anchor Kent Brockman is briefly under the mistaken impression that a "master race of giant space ants" is about to take over Earth. "I, for one, welcome our new insect overlords," Brockman says, sucking up to the new bosses. "I'd like to remind them that as a trusted TV personality, I can be helpful in rounding up others to toil in their underground sugar caves."
Even if you're not intimately familiar with that episode (and you really should be), you might have come across the "Overlord Meme," which uses Brockman's line as a template to make a sarcastic statement of submission: "I, for one, welcome our (new) ___ overlord(s)." Over on Language Log, where I'm a contributor, we'd call this kind of phrasal template a "snowclone," and that one's been on our radar since 2004. So it's a repurposed pop-culture reference wrapped in several layers of irony.
But what would Watson make of this smart-alecky remark? The question-answering algorithms that IBM developed to allow Watson to compete on Jeopardy! might lead it to conjecture that it has something to do with The Simpsons -- since the full text of Wikipedia is among its 15 terabytes of reference data, and the Kent Brockman page explains the Overlord Meme. After all, Watson's mechanical thumb had beaten Ken and Brad's real ones to the buzzer on a Simpsons clue earlier in the game (identifying the show as the home of Itchy and Scratchy). But beyond its Simpsonian pedigree, this complex use of language would be entirely opaque to Watson. Humans, on the other hand, have no problem identifying how such a snowclone works, appreciating its humorous resonances, and constructing new variations on the theme.
All of this is to say that while Ken and Brad lost the battle, Team Carbon is still winning the language war against Team Silicon. The "war" metaphor, incidentally, had been playing out for weeks, stoked by IBM and Jeopardy! to build public interest in the tournament. The press gladly played along, supplying headlines like the one in the Science Times from Tuesday, "A Fight to Win the Future: Computers vs. Humans." IBM knew from the Kasparov vs. Deep Blue days that we're all suckers for the "man vs. machine" trope, going back to John Henry's mythical race against the steam-powered hammer. It certainly makes for a better storyline than, say, "Check out the latest incremental innovations that Natural Language Processing researchers are making in the field of question-answering!"
I first encountered IBM's hype about the tournament last month, during the NFL's conference championship games, when Dave Ferrucci, the ebullient lead engineer on the project, showed up in commercial breaks to tell us about the marvels of Watson. One commercial intriguingly opens with Groucho Marx telling his classic joke: "One morning I shot an elephant in my pajamas. How he got into my pajamas, I don't know." Ferrucci then begins: "Real language is filled with nuance, slang and metaphor. It's more than half the world's data. But computers couldn't understand it." He continues: "Watson is a computer that uncovers meaning in our language, and pinpoints the right answer, instantly. It uses deep analytics to answer questions computers never could before, even the ones on Jeopardy!" Then a Jeopardy! clue is displayed: "Groucho quipped, 'One morning I shot' this 'in my pajamas.'"
Now, that's a provocative set of claims. Watson's performance in the tournament (despite a few howlers along the way) clearly demonstrates that it is very skilled in particular types of question-answering, and I have no doubt it could handle that Groucho clue with aplomb. But does that mean that Watson "understands" the "nuance, slang and metaphor" of natural language? That it "uncovers meaning in our language"? Depends what you mean by "meaning," and how you understand "understanding."
Elsewhere, Ferrucci has been more circumspect about Watson's level of "understanding." In an interview with IBM's own magazine ForwardView, he said, "For a computer, there is no connection from words to human experience and human cognition. The words are just symbols to the computer. How does it know what they really mean?" In other words, for all of the impressive NLP programming that has gone into Watson, the computer is unable to penetrate the semantics of language, or comprehend how meanings of words are shot through with allusions to human culture and the experience of daily life.
Such a sober assessment doesn't jibe with popular perceptions of artificial intelligence, of course. We want our "smart computers" to engage with us linguistically like HAL did in Stanley Kubrick's 2001 -- well, except for the murderous rampage part. (Ferrucci and his team prefer to use a different pop-cultural point of reference: the original series of Star Trek, with its more benign on-board talking computer.) But let's remember how HAL was introduced, via a BBC interview watched by the spaceship crew in the film:
The sixth member of the Discovery crew was not concerned about the problems of hibernation. For he was the latest result in machine intelligence -- the HAL 9000 computer, which can reproduce, though some experts still prefer to use the word 'mimic,' most of the activities of the human brain, and with incalculably greater speed and reliability.
Is Watson, despite its limitations, nonetheless a precursor to a HAL-esque machine that can "mimic" natural language, if not "reproduce" it? Baby steps, baby steps. First we need a computer that doesn't give Toronto as an answer to a clue about "U.S. Cities," as Watson memorably did for Final Jeopardy in the first game. And we'd also want it to know that the "anatomical oddity" of Olympian gymnast George Eyser was not his leg, but his missing leg.
Those were two isolated gaffes in a pretty clean run by Watson against his human foes, but they'll certainly be remembered at IBM. For proof, see Stephen Baker's book Final Jeopardy, an engaging inside look at the Watson project, culminating with the Jeopardy! showdown in the final chapter. (In a shrewd marketing move, the book was available electronically without its final chapter before the match, and then the ending was given to readers as an update immediately after the conclusion of the tournament.) Baker writes:
As this question-answering technology expands from its quiz show roots into the rest of our lives, engineers at IBM and elsewhere must sharpen its understanding of contextual language. Smarter machines will not call Toronto a U.S. city, and they will recognize the word "missing" as the salient fact in any discussion of George Eyser's leg. Watson represents merely a step in the development of smart machines. Its answering prowess, so formidable on a winter afternoon in 2011, will no doubt seem quaint in a surprisingly short time.
Baker's undoubtedly right about that, but we're still dealing with the limited task of question-answering, not anything even vaguely approaching full-fledged comprehension of natural language, with all of its "nuance, slang and metaphor." If Watson had chuckled at that "computer overlords" jab, then I'd be a little worried.
We want to hear what you think about this article. Submit a letter to the editor or write to firstname.lastname@example.org.