Right now, the team’s work has shipped in small ways, like the birthday sticker trick. But that is just the beginning of a development program that wants to transform the way you use the camera on your phone.
AI Camera combines many of the most important technological developments of the last several decades: neural networks, robotics, camera systems, and social-network data. This underlying basket of technologies—more adjacent to each other than in “a stack,” as software developers might conceive it—are converging into the smartphone's ability to take and display pictures.
Perhaps that seems absurd. But the human desire to capture, understand, and share images of the physical world has proven to be nearly insatiable, which is why this is the one domain where Facebook, Apple, Google, Samsung, Snapchat, and Microsoft directly compete.
Facebook’s work mirrors what’s happening at the other tech giants. Snap calls itself a camera company, and its realization of “lenses” are the best embodiment of augmented reality outside of Pokémon Go. At Google’s developer conference in May, Sundar Pichai showed off Google Lens, software that can detect what a camera is seeing and do something with that information, from entering a password to identifying a flower.
Prodded by Snap, the tech giants have begun to piece together what can be accomplished with the whole imaging and display system that a smartphone is. Every millisecond a phone’s camera is engaged is a moment when data can be captured, processed, understood, and looped back to the user for viewing.
“We’re basically looking at what pieces of technology we need to build amazing augmented-reality products,” said John Barnett, product manager on the AI Camera team.
Imagine, he said, a persistent, shareable social layer on the physical world, a spatialized Facebook that’s escaped the feed.
“Everyone got so excited about Pokémon Go when it was just one thing. What if there are 1,000 things like that?” Barnett asked. “All these layers of information that are spatially situated and relevant to what you care about.”
This is a radically different notion from the Facebook we’ve come to know, which, even though it made the leap from the desktop to “mobile,” rarely engages with the physical space where your hand clutches the phone.
“In the existing Facebook structure, we’re giving you everything that’s happening right now in the world, collapsing space to give you a a slice of time,” said Barnett. “This is talking about collapsing time to give you this piece of space.”
Facebook would take on two modes: The News Feed, in the company’s terms, would show you what you care about now, and the spatial Facebook would tell you what’s happening here. One could read from, and write to, the world. Your world, at least.
* * *
On one of the many decks at Facebook’s Menlo Park campus, overlooking the mudflats of the south Bay, there’s a nondescript corner. Pipes run along it. A surveillance camera sits on the east-facing wall. To the naked eye, there’s nothing to distinguish it from the hundreds of others that help form Facebook’s gargantuan ark.