On a Friday night in New York City you can find just about anything. And this past Friday about 130 hackers gathered in the Hayden Planetarium to participate in the American Museum of Natural History’s very first hackathon.
The premise was simple: The museum handed the huge dataset they call The Digital Universe to the hackers and gave them 24 hours to make something. (Part of what made this hackathon different was the literal universe of data hackers were given. More on that in a minute.) There were some specific challenges and categories (Education, Visualization, Tool Kit, and Wildcard) but the hackers were otherwise free to explore the data and run with it.
Hackathons are often most useful to the hackers themselves—participants come and work through ideas, meet one another, and learn new tricks and coding languages. They’re great events for community building, for publicity and for experimentation, but rarely—in my experience—do hackathons create lasting, useful products. This hackathon was different.
It shouldn't have been. The hackers had a lot working against them. The dataset the museum threw at them is huge. The Digital Universe combines data from dozens of different organizations into a three-dimensional atlas of the entire Universe. Before digging into the data, you have to sift through pages and pages of documentation. Partiview, the public software that allows for people to fly through the data, is difficult to use. And not only is it a lot of data, it's really difficult data. It involves measurements of stars and exoplanets and satellites that are billions of miles away, spanning literally the entire universe. The majority of these hackers had no training in astronomy, and not all of them were skilled at data analysis. So they faced two overlapping challenges: They had to wrangle a huge dataset, and they had to wrangle a huge concept.