Imagine being told the storm of the century was going to hit in three days--anywhere between New York City and Richmond, Virginia. That may have been the scenario when superstorm Sandy turned toward the East Coast last month.

Thankfully, five years ago the National Hurricane Center (NHC) began utilizing the Hurricane Weather Research and Forecasting model that analyzes big data collected from satellites and airborne observations, producing high-resolution computer-modeled forecasts every six hours. As a result, the NHC was able to predict the storm's landfall five days out within 30 miles of direct impact in New Jersey. (Twenty years ago, even 72-hour forecasts were only accurate up to about 350 miles of landfall.) Big data was more than just a buzz word during Hurricane Sandy; it saved lives.

This is one clear example of the growing role of big data in the public sector. From election campaigning to setting NASA's budget to prepping for the next big natural disaster, the information derived from analysis of large data sets is being used to make government agencies more effective Back in March, the White House announced its "Big Data Research and Development Initiative", aimed at creating efficiencies, reducing waste, and powering the economy. Six federal departments and agencies committed more than $200 million to the development of new tools and techniques for accessing, organizing, and gleaning discoveries from huge volumes of digital data.

TechAmerica, an advocacy group for the U.S. tech industry, produced a groundbreaking report, "Demystifying Big Data: A Practical Guide to Transforming the Business of Government", as part of this initiative. While the report asserts that big data could have an impact as profound as the Internet, it will be up to the government to leverage the experience and skills of all sectors of society to put this data to proper use.

"By utilizing the advanced techniques and technologies many agencies already have in place, agencies can and must harness the volume, velocity, and variety of data at its disposal to provide citizen services more efficiently and at less cost," said Chris Wilson, vice president and counsel of TechAmerica and also staff director of the group's Big Data Commission.
In partnership with the National Science Foundation and the Department of Energy, NASA recently launched a major initiative of its own: Big Data Challenge, a series of competitions designed to develop new approaches to using big data information sets from various U.S. government agencies. The competition drew 16 submissions, with data supplied from agencies such as the Department of Energy, the Department of the Interior, the Department of Health and Human Services, and the National Oceanic and Atmospheric Administration.

One of the goals of the project is to solve a problem many government agencies face: how to pry useful information from past data in disparate formats. This is an especially intimidating problem at NASA. The space agency regularly engages in missions where data is continually streaming from orbiting spacecrafts faster than it can be stored, managed, or interpreted. NASA is currently planning missions that will stream more than 24 terabytes every day, roughly 2.4 times the data stored within the entire Library of Congress print collection.

It's no wonder that big data is such a top-of-mind issue for government agencies. It may surprise you to know that big data is also being used to foster civic responsibility. Tech innovators from across the country are leveraging big data to empower citizens and increase transparency in government.

"We're trying to use data to improve civic engagement and are teaming with Code for America and the New York City government on a couple of projects to try to get people out there and involved," said Jake Porway, founder and executive director of DataKind. "Getting people to understand the vast amount of data now publicly available through sites like data.gov is a way of increasing civic literacy." DataKind, which matches volunteer data analysts with social organizations, is one of the nonprofits trying to use big data to cure some of the world's ills and inspire civic participation for social good.

Code for America is another nonprofit leveraging technology and online resources-- including the breakdown of big data--to empower citizens and get residents involved in civic endeavors to help out cash-strapped cities and local governments. The group offers fellowships each year to those interested in using the Internet to make governments more open and efficient. In addition to a small stipend, fellows receive training and support from seasoned web pros.
The group's accomplishments include systems to track service requests in cities, an adopt-a-hydrant program in Chicago (where people take part in the upkeep of hydrants), and even a program that puts residents in charge of battery checks for Honolulu's tsunami warning sirens.

Nicholas Doiron, a 2012 fellow at Code for America, took part in a project to catalog all housing cases-- disputes reported to the housing department, such as landlord disputes or criminal investigations-- in Macon, GA. The data was used by a community group to track neighborhood improvements and declines, as well as by Habitat for Humanity to identify housing cases not reported to the city. "We made this data available to the community, as well as the housing department, for the first time in a long time," Doiron said.

It's clear that big data isn't just an abstract concept--it has myriad practical applications. In the aftermath of Sandy, international nonprofit Direct Relief, an emergency response organization, used analytic and data visualization tools from Palo Alto-based Palantir Technologies to pinpoint clinics located in flood-risk zones near Sandy's path. These tools have allowed the organization to pull together massive amounts of information into a common framework to better understand and plan for emergencies in real-time. As government agencies and civic-minded organizations become increasingly adept at handling big data, citizens are the true beneficiaries.