Machine Learning: Building "The Smartest Machine on Earth"

PBS's NOVA program re-aired a very interesting episode last night titled "The Smartest Machine on Earth", which shows the development of IBM's "Watson" supercomputer, built by IBM researchers to play the television game show Jeopardy! at a level equal to, or better than, the best human players. 

While this is a fascinating (and funny) story in its own right, this program is of special interest to Proofpoint fans because it gives a pretty good explanation of how machine learning techniques can be used to solve what, for computers, are very difficult problems — you know, things like understanding a Jeopardy question or telling the difference between spam and legitimate email.

The problem described so well in this show is exactly analogous to the problem of detecting spam: There's no way to write enough rules to create a computer program robust enough to understand (much less accurately answer) the vast diversity of Jeopardy questions, just as writing rules to detect spam becomes ineffective very quickly. So you need to develop a system that can learn, more-or-less on its own, how to accurately solve the problem. These same sorts of techniques are at the heart of Proofpoint MLX anti-spam technology.

The parts of the show that describe the machine learning techniques used by Watson are in the second half of the full show. The embedded video from PBS, below, gives only the first 20 minutes of the program... But if you're a geek like me, you'll be hooked a few minutes in.

Watch the full episode. See more NOVA.