Connecting the Dots: Finding Patterns in Large Piles of Numbers

A new program can find and compare relationships in complicated data without having to be asked specific queries

connecthedots-body.jpg

Are there subtle patterns lurking in data that can foretell of a coming financial-system crash? What can explain the variations in sports-star salaries? How about the complex relationship between genes and certain diseases? Scientists in various fields have been searching for better ways to analyze large piles of data for such patterns, but the difficulty has always been that they need to know what they're looking for in order to find. A new software program, described in the latest issue of Science, is designed to find the patterns in data that scientists don't know to look for.

David Reshef, one of the scientists behind MINE, as the program is called, explains, "Standard methods will see one pattern as signal and others as noise. There can potentially be a variety of different types of relationships in a given data set. What's exciting about our method is that it looks for any type of clear structure within the data, attempting to find all of them. ... This ability to search for patterns in an equitable way offers tremendous exploratory potential in terms of searching for patterns without having to know ahead of time what to search for." MINE compares different possible relationships (including linear, exponential, and periodic)  and returns those that are strongest.

On MINE's website, the program is available for download.


Image: kentoh/Shutterstock.

Presented by

Rebecca J. Rosen is a senior editor at The Atlantic, where she oversees the Business Channel. She was previously an associate editor at The Wilson Quarterly.

Adults Who Live In Treehouses

A treehouse can be an ideal office space, vacation rental, and way of reconnecting with your youth.

Join the Discussion

After you comment, click Post. If you’re not already logged in you will be asked to log in or register.

blog comments powered by Disqus

Video

The Best 71-Second Animation You'll Watch Today

A rock monster tries to save a village from destruction.

Video

The Case for Napping at Work

Most Americans don't get enough sleep. More and more employers are trying to help address that.

Video

A Four-Dimensional Tour of Boston

In this groundbreaking video, time moves at multiple speeds within a single frame.

Video

Who Made Pop Music So Repetitive? You Did.

If pop music is too homogenous, that's because listeners want it that way.

Video

Stunning GoPro Footage of a Wildfire

In the field with America’s elite Native American firefighting crew

More in Technology

Just In