Complex financial information is hidden in plain sight all over the planet, according to James Crawford, CEO of Orbital Insight. The number of ships docked at a Malaysian port, even the color of a wheat field in western Nebraska, are actually signs, Crawford explained to me, visible indicators of economic activity, not just for a local region but for an entire global industry.
Seen this way, mundane landscapes previously deemed unworthy of analysis can, in fact, be meticulously—and profitably—scrutinized. This newfound appreciation is not aesthetic, of course, but fiscal, as even the growing shadows of a Chinese construction site can be interpreted as valuable clues about the strength of the underlying economy.
Crawford’s company, Orbital Insight, is one of a new breed of market-research firms pioneering the use of high-resolution satellite imagery. This is called geo-analytics, or geography crossed with the algorithmic firepower of Big Data. With access to satellite images—refreshed on a daily basis and available at a scale of one meter per pixel—companies such as Orbital Insight use artificially intelligent deep-learning algorithms to sort through the data and look for patterns.
Often this is just about change-detection: that is, looking for a particular pixel that has flipped from one color to another, thus indicating a new agricultural condition or the beginning of major construction work. Other times, it is all about quantity: methodically counting the number of cars parked outside a shopping mall in Minneapolis, or the trucks lined up outside a Chinese steel yard. In either case, it’s about combining machine vision with data science, or giving computational power a large enough visual dataset to work with.
Visual evidence captured by satellites is thus now subject to narrative interpretation for the purpose of extracting potential financial insight—and this potential financial insight can then be sold to paying customers. This, in fact, is Orbital Insight’s operating business model, marketing its geo-analytic expertise to hedge funds, U.S. government agencies, and nonprofits alike for what those groups might be able to learn from satellite data. Nonprofits, for example, might look for the level of water in a remote desert lake or threatened reservoir, or the true extent of a developing city as it sprawls into a nearby national park, for indications of what policies they might next pursue; hedge funds might see this same data and decide to short certain commodities.
Crawford’s own background here is instructive for understanding what is really happening: he was once engineering director of Google’s book-scanning project, another daunting task in which huge amounts of visual data had to be sorted by machines so that tiny marks on a page could be recognized as letters, then those letters as words, then those words as searchable sentences. “It was a similar problem,” he explained to me. “It’s just a tremendous number of pictures. You have to figure out where the text is and what it all means; you have to put it into Google search and so on.”
The challenge is now in applying this same form of graphic analysis to landscape: learning to read the surface of the Earth such that topographic features, building shapes, and the presence of shadows can all begin to reveal their deeper meanings. It is, in other words, about transforming the world into a hieroglyph—and then cracking this new terrestrial language, making it legible not only by human beings but by semi-intelligent machines. Not one to mince words, Crawford refers to this as developing “new understandings of the Earth.”
Speaking to Crawford as he reels off future applications of his firm’s technology is like listening to a one-man litany of geographic features and their potential implications. He is a reader in search of a text. Crawford will spin through long and fluid lists of everything from how many houses are going up in the suburbs of Phoenix to the number of train cars clacking across a Chinese railroad network—to the perceived attendance of suburban churches in the U.S. and the corn yields of large-scale agribusiness in the American heartland to rice production in southeast Asia.
“We view the Earth as really big and really complicated—but fundamentally understandable,” he explained. “Using this combination of AI technology and humans, working in partnership, we can handle that volume of imagery to extract the insights that might be implicit in it. The question we ask ourselves is: where are the information gaps about the Earth?”
Rather than character recognition, then, it is what we might call landscape recognition.
* * *
At the end of May 2016, a small team led by Golan Levin at Carnegie Mellon University released a new online tool called Terrapattern. Terrapattern, as The New Yorker described it, can be thought of as “Google Earth’s missing search engine.” It is capable of scanning—and automatically finding visual similarities amongst—thousands of satellite images of the Earth’s surface. You can click on any visible feature—Brooklyn rooftops, Detroit parks—and Terrapattern will instantly locate any other nearby landscapes that resemble it. It will, in other words, match the pattern.
In essence, Terrapattern simply gives us—the general public—a glimpse of some of the proprietary analytic tools that firms such as Orbital Insight already have at their disposal. For now, Terrapattern’s geographic resourcefulness is limited to Pittsburgh, San Francisco, Detroit, Berlin, and New York City, but its goal is global: an entire planet subject to autonomous visual analysis. A parking lot in Knoxville, a football field in Fresno, a factory complex near Salt Lake City: these organized landscapes will eventually find their visual twins in places as far afield as South Africa, Venezuela, and Japan.
It is intriguing to consider how this interpretive process might be flipped on its head, so to speak, or turned around from back to front, to look at how these machine-learning algorithms might be made to fail in intriguing ways. Doing so would seem to suggest a range of peculiar new landscape design opportunities, both for governments hoping to maintain secrecy and for private firms wishing to disguise their everyday activities.
A Walmart hoping to look better-attended than it really is, for example, could spoof the satellites with a parking lot full of fake cars, or a new residential building could be playfully designed so that its roofscape looks like a neighborhood park, throwing off the watchful eyes of Terrapattern.
Consider the work of artist Adam Harvey. Harvey has made a name for himself exploring, among other things, how cosmetics might be used “to break apart the continuity of a face,” in his words, so that that person can no longer be identified by facial-recognition algorithms. He calls this “camouflage from face-detection technology.”
There is no reason to believe this sort of effect could not be scaled up to the level of architectural or landscape design, like dazzle paintings on the hulls of World War I ships, deliberately employed as a kind of corporate smokescreen to evade financial interpretation from above. Duping the algorithm, so to speak—or disrupting the hieroglyph—suggests a way to subvert the era of total visibility that Orbital Insight and Terrapattern both imply.
Those eyes soaring high above the clouds might always be watching us, in other words, but they needn’t always understand what they see.
We want to hear what you think. Submit a letter to the editor or write to letters@theatlantic.com.