An article on Grantland explored how the Toronto Raptors have used the SportVU system to build a sophisticated model of their team’s play. Their model not only detects what kind of play is happening—a pick-and-roll, say—but compares the team’s players in every defensive play with theoretical “ghost players” whose movements minimize their opponent’s expected point value. The team learned, for instance, that the ghost players were “consistently more aggressive on help defense than the real Toronto players”—a finding that translated into actionable coaching advice.
It’s an entirely new kind of vision. We don’t often think of computation that way, as a visual aid, because it’s somewhat difficult to describe what it helps us see. Where telescopes and microscopes show us the very far and the very small, the computer shows us the very much, all at once. It makes time available to the mind and eye. Computation, in that sense, is a kind of compacting of imagination: It helps us generate and explore a zillion scenarios and digest them into a representation that’s easy to play around with.
* * *
Forty years ago, one of the most common surgical procedures in North America was called an “exploratory laparotomy,” which was a fancy name for, “Let’s open the patient’s abdomen and see what we can find.” Today, doctors don’t go exploring with a knife—they use things like CT scans, ultrasounds, and MRIs.
Modern medicine has in some sense been built on the back of better ways of seeing. The X-ray let us see bone through skin, which took the guesswork out of treating fractures and let us detect early tumors. CTs and ultrasounds let us see organs, blood vessels, muscles, and other soft tissues in three dimensions, which caused a revolution in diagnostic medicine and made surgery radically more precise and safe.
CT was made possible by the computer, which stitches together a collection of X-rays into a reconstructed 3-D image. But this is still more or less a static enterprise: a CT study is more like a picture than a movie. What if you could do for medicine what we’ve already done for chess and basketball—what if you could somehow use the computer to see not just what’s there, but what could be?
In some specialties, this is already becoming possible. Radiation oncologists, for instance, use accelerated beams of radioactive particles to destroy cancers. It used to be that these beams were targeted somewhat crudely: You’d take a two-dimensional X-ray of your patient and outline the area you wanted to zap (the tumor) and the areas you wanted to avoid (healthy organs). Since X-rays couldn’t show you much in the way of soft tissue, you had to use nearby bones as landmarks.
Today, radiation treatments are planned using software. The doctor identifies tumorous and healthy tissues in slice after slice of a CT scan by drawing on the slices directly, on the computer, as though coloring in a figure in MS Paint. This creates three-dimensional contour maps of the tumor and nearby organs. The software then takes these contours and runs hundreds of thousands of simulated treatments against them, using a model of how radioactive particles will behave in different tissue types—how they’ll be absorbed, how they’re likely to ricochet, and so on—to determine the ideal angle and power settings of the real beam.