In the old days, football offenses were simple. Hand the ball to a strong, fast guy with stronger guys in front of him, evade tacklers. But the game has evolved. Schemes meet counterschemes, morph, adapt. To watch the Oregon Ducks play football is to realize that sports, like the stock market and smartphones, have become almost too complex to understand. The way games are broadcast has changed, too. What you see on television is no longer a faithful reproduction of the view from the 50-yard line; now cameras hang from blimps, zip over the tops of players’ heads, zoom in on their toes. Slow-motion replay has been written into the rules. We have cyborg referees.
Take the yellow stripe that’s virtually superimposed onto a football field to mark the first-down line. It’s an example of what nerds call augmented reality, in which a digital representation of the physical world is overlaid on live video. It also speaks to a larger phenomenon: the way data-driven technology is changing the way the game is played and viewed. Here, Hank Adams, the CEO of Sportvision, the company that created the yellow line, talks about how fans will experience sports in the future. Hint: you won’t be just watching.
ALEXIS MADRIGAL: How does the yellow line work? I’ve always wondered.
HANK ADAMS: There are a couple of systems that work in concert. Sensors on the cameras tell us where a camera’s looking, and measure how it’s pointed—the pan, tilt, and zoom. We also have a virtual model of what the field should look like. We paint the line there. Then as the camera pans, tilts, and zooms around, we map the three-dimensional world onto the two-dimensional video.
AM: How do you draw the line just on the field, without having it paint over the players’ feet or anything?
HA: Think of the green screen of a weatherman—but next-generation technology. We look for the color of the grass in the place we want to draw the line, and anytime we don’t see it, we presume there to be either some logo on the field or a player’s shoe or pants. When you have Green Bay Packers jerseys, or something that’s very close to the color of the grass, it’s very difficult to distinguish between the players and the field, color-wise.
AM: Why hasn’t ESPN done this itself?
HA: Our company evolved and spun out of News Corp. Back in the day, Fox got the rights to the NHL. Because hockey’s such a fast game, it can be hard to follow the action. And so David Hill, a visionary sports executive, said, “I want some way to actually follow this damn puck.” He pulled together some of the smartest scientists and technical minds that existed across all of News Corp, put them on this project, and lo and behold, they figured out how to make the puck flash with infrared emitters—you wouldn’t see it flashing in person, but the camera could see it. The glowing puck was the first example of augmented reality, to my knowledge, that had ever been seen by people.
When that came out, ratings went up 40 percent. Letterman was doing skits on it, Labatt beer commercials were spoofing it—the glowing puck was everywhere. Fox said, “This is great, but we can’t afford to keep up this traveling circus”—a truckload of equipment for every game, with very expensive engineers. So our company was born.
AM: Then you had the big hit with the first-down line. Did any of your early experiments fail?
HA: We measured the distance of home-run balls—that was an expensive technology that proved not all that insightful. It actually doesn’t really matter how far the ball goes. We measured how high basketball players jump, thinking this would illustrate their athleticism—until we discovered that on average they’re jumping about 12 inches. Turns out they only jump as high as they really need to.
The next success was the K-Zone for baseball, which allowed you to see whether pitches were in the strike zone. Around that time, we were also asked to do NASCAR racing and track cars. That’s when the lightbulb went off. We realized, wait a sec, we’re capturing a digital record that’s fundamentally different than a video record: We have a three-dimensional, real-time computer model of what’s happening on the NASCAR track. A video of a race shows maybe a handful of cars at any given time. But there are 43 cars on the track, covering 2.5 miles, so what you see on TV is the minority of what’s going on at the track. Because we’re tracking every car all the time, I can provide precise data on how a car is cornering or whether it clipped a wall. We can tell you things that you can’t see.That’s where digital records start to get interesting. Ultimately, I think, we’re creating the next generation of sports entertainment. I can let you hit the pitch with a motion-sensing game console like the Xbox Kinect [which users control via gestures and spoken commands], because I can replicate exactly how the ball moved.
AM: Have you put any of this into practice yet, or is it just a big idea?
HA: We already have a product called RaceView, which lets you watch the NASCAR race through a video-game interface. You follow your favorite driver the whole race, and as you watch, you can spin the track around, pinch, zoom, or pick different cars. This is not the same as watching video.
AM: Do you imagine these sorts of products for other sports?
HA: Once we’ve got what are generally called connected TVs—smart devices like Samsung is making, or an Xbox, a PlayStation, an Apple TV—there’s an awful lot we can do. With great rendering and graphics, with the Kinect interface, these are very, very sophisticated computers that are going to change the sports experience. I’m going to be able to interact, predict, and compete against my friends in ways that are much more involved than sitting back and watching a three-hour broadcast.
AM: So are you in the entertainment business, or the data business?
HA: I’m really trying to develop our sporting experience, but we call ourselves a data company. We sell the data for player development, analytics, coaching, drafting.
AM: Which sports are using it most?
HA: Baseball—it’s an arms race. I think the Yankees have half a dozen data scientists on their staff. All dealing with, basically, our data.
AM: What’s the hardest sport to get data on?
HA: Computer vision, using cameras, works when players behave relatively well—that is to say, they don’t collide with each other a lot. That’d be baseball. But the advantage of players colliding is that they wear gear. And I can put tracking systems on players in shoulder pads and helmets. Baseball players just don’t have that protection; all you can do is use computer vision and algorithms. Basketball is tricky too.
NASCAR’s our most complicated tracking system. We have loads of room to get stuff into the car—our tracking system, with batteries, is five pounds or so—and we track to two-centimeter accuracy. Data makes NASCAR more interesting. It really is a technical sport, where strategy matters. You can find out things like how big of an impact the pit crews have on performance and outcome—turns out, a very big impact. That kind of stuff can help tell the story: When do you do your pit stops? Do you take two tires, do you take four tires? How did that impact the race? Are you slipping? We can accurately predict when somebody’s going to overtake somebody else.
AM: How do other sports use the data?
HA: When coaches see our data laid over video, they go crazy. They have never seen the data before, because their players have never been tracked. They can explain to a player, “Hey, you can’t give him this much room; look what happens every time you give him more than five yards.” This huge database can show that if you give a certain receiver more than X amount of room, he’s going to do Y. When you have a digital record of the event, you have it all.