When people talk about Big Data, they often talk in terms of Messianic solutions for economy-size systems. Roads and highways. Diagnoses and treatments. Buyers and sellers.
But some of the most interesting work being done with data addresses a different kind of complex system. The information for this system is neither private nor proprietary: In fact, its ownership is a little more evergreen.
It’s the weather. And this week, one of the more interesting recent online weather data products opened to the public and explained itself.
It’s called Quicksilver. Quicksilver aims to provide the highest-resolution, most up-to-date map of global temperatures ever created. Click around its maps or zoom in, and it paints hot reds, frigid blues, and temperate greens at a more detailed, more local level than any previous planetary* temperature map ever has.
It does all this without adding any new sensors to the world: Humanity’s raw observational power wasn’t increased to make the Quicksilver map work. Rather, the Quicksilver team merged and correlated existing data, from different public sources, for the map.
A blog post, just posted on the Quicksilver website, explains how they did it. The map relies on temperature data from a number of sources. One of those is the Global Forecast System (GFS), a service of the U.S. National Oceanic and Atmospheric Administration. GFS measurements are “live,” or updated so frequently as to be essentially live, but they're at one-tenth the resolution of the Quicksilver map.
Meanwhile, much higher-resolution temperature information exists, but it isn’t live. Sources for this kind of data are NASA’s two MODIS satellites. Those structures take incredibly detailed readings of the Earth’s surface temperature, in addition to a number of other readings, like how much heat an area emits.
The Quicksilver team, members of the Dark Sky Company, figured out how to put all those different data together and compute “microclimates,” the small, environmental variations in climate that GFS doesn’t pick up. GFS, for example, might report the temperature of a large area as 72 degrees Fahrenheit, but if more detailed data suggest that the area on top of a hill within that area is usually 3 degrees cooler* than the rest of it, Quicksilver will adjust for that hilltop on its map.
Quicksilver, in other words, is a map of estimation, of global guesswork.
Quicksilver’s blog post explains the entire process better than I can, but two things strike me as especially remarkable about the project. The first is the beauty of these maps: In their fluctuations and spinning pressure systems, they give us a way to imagine the temperature fluctuations of the whole Earth. They are evocative in a high sense.
Second, I’m impressed by the obvious concern this team has for history. This concern runs through all the team’s products. It’s best known for its app, Dark Sky, which tells a user if it’s raining now over her exact current location. Its second app, though, Forecast.io, aspired to be a kind of universal weather data provider, a clearinghouse not just for current conditions but also for weather data from as far back as feasible. Forecast.io also included an API for accessing historical data.
Quicksilver has that capacity and more. It includes an API, but it’s a considerably easier-to-use API than most. Instead of accessing it through a text-only Terminal interface, for Quicksilver’s API, you can enter a URL into your web browser. A URL on this pattern --
-- will return a large TIFF for a certain period of time. So a map of global temperatures for August 28, 2013, say, at 3 p.m., would require the substitution of “2013” for “YEAR,” “08” for “MONTH,” etc., for a URL like:
(That “15” at the end of the URL is the Coordinated Universal Time. For 3 p.m. on, e.g., the American east coast, times would have to be seasonally adjusted to Eastern Standard or Eastern Daylight Savings time.)*
An API like this makes Quicksilver’s data accessible to folks who can’t, or don’t think they can, code. That’s perhaps a bigger deal than Quicksilver’s guesswork, a move that’s not only generous, but also smart. It ensures that many, many more folks will use the product and the API than if it were accessible only through coding.
Quicksilver may be a small product, but it’s a useful, thoughtful, and evocative one. It not only helps us use the data that already exists; it provides one more tool to understand the whole Earth.
* An earlier version of this article made it seem as though Quicksilver is the most-detailed temperature map ever, though it is only the most-detailed global version. The current version also better explains microclimates (it is more likely to be cooler on a hill, not warmer, as originally stated) and UTC.
This article available online at: