Skip to main content

Why Watson's win doesn't make humanity obsolete -- yet

Despite Watson's tremendous performance, it struggles with simple questions most humans can answer easily.
Despite Watson's tremendous performance, it struggles with simple questions most humans can answer easily.
  • IBM's Watson computer doesn't really "think" anything; it struggles with simple questions
  • Most of the clues on "Jeopardy!" mention proper nouns -- specific places, events, people
  • Watson's eventual commercial incarnation will be as a tool, not a human replacement

(CNN) -- Well humans, it's been a good ride, but after being eviscerated by IBM's supercomputer Watson on "Jeopardy!," it's probably time to pack up the truck and let the machines inherit the Earth.

Or is it?

Despite Watson's tremendous performance, the Final Jeopardy question at the end of Tuesday night's airing revealed the Achilles' heel that computer scientists have known all along: Watson doesn't really "think" anything, and it struggles with simple questions that most humans can answer without a second thought.

Most of the clues on the "Jeopardy!" board mention proper nouns -- specific places, events, people, songs, books and so on, says Dr. Douglas Lenat, a machine learning pioneer, former Stanford professor of computer science and CEO of Cycorp, a company that develops semantic technologies.

"This gives the Watson algorithm a great deal of 'traction.' To us viewing the show, it's impressive if it correctly knows that Franz Schubert's birth date was January 31, 1797. But if that date had been part of the clue, could Watson correctly pick out [Schubert's] maternal grandmother's birth date from a list where only one of the dates was earlier than 1797?"

'Jeopardy' contestant versus computer
Watson gets a job

We could, because we understand that everyone is younger than their own mother and grandmother, but Watson is unable to understand this, Lenat explained.

At the end of the day, Watson is not really conceptualizing a clue's meaning. It simply number-crunches its way to the right answers by comparing vast amounts of data. This is why it dominates the "fill in the blank" knowledge clues (Aeolic, spoken in ancient times, was a dialect of this), but falters on some more "common sense" deductions.

The biggest blunder was in the first game's Final Jeopardy round.

Lenat broke down IBM's explanation of why Watson guessed "Toronto?" to the clue in the category of "U.S. Cities:" "Namely, the string 'U.S.' often appears in articles near the word 'America,' which appears often near the phrases 'North American' and 'American League,' and so on, all the way to the word 'Toronto.' "

As Dave Ferrucci of IBM Research says, this points out the limitations of relying so heavily on statistical reasoning. The result, as in Watson's case, is akin to a human autistic savant."

"Jeopardy!" is challenging because the clues are esoteric. No one would watch a battle of common sense.

But Lenat notes that a question like "If a snowman melts and later refreezes, does it turn back into a snowman?" would be nearly impossible for a statistical reasoning program to tackle. You can't answer it by calculating how many times "snowman" appears next to "melt" and "refreeze" in every article and book ever written.

Not so hot now, are you Watson?

Last week, I interviewed Stephen Baker, a journalist with unfettered access to Watson's development at IBM. I asked him about people's concern that computers with natural language capabilities could replace members of the work force.

"People have good reason to worry about machines supplanting them," Baker said. "That's what technology does. Tractors, forklifts, word-processing software, they all took away jobs. And people, with their creative minds, have used them throughout history to figure out where the next jobs will be."

And like its forebears, Watson's eventual commercial incarnation will be a tool, not a human replacement. Lenat thinks of Watson or Deep Blue or even the steam-powered hammer that bested John Henry in American folklore as an "impressive demonstration that a certain task can, despite popular intuition at the time, be performed better by a machine."

Watson's natural language abilities might make it a good candidate for work in call centers, where it could interpret customers' needs and direct them to the right information (or people) via conversation.

There's also been discussion of how Watson could be used in medicine to cross reference a patient's symptoms with compendiums of digitized medical journals and data -- a sort of statistical second opinion to a doctor's professional experience.

But because Watson is only a "statistical brain," and not an analytical one, don't expect to see it managing your HR department or running a daycare facility anytime soon.

We'll still be getting up early and trudging off to do those jobs for the foreseeable future. Thanks for nothing, IBM.

Matt Silverman is an editor at Mashable. The opinions expressed in this commentary are his alone.


Most popular Tech stories right now