It's The Computer Code, Silly


In trying to unravel what the real story is behind climategate - not the hysterical conspiracy theories but the skeptic's valid concerns about corrupted data and the bias of groupthink - John Tierney is, as always, worth reading. But more and more readers and bloggers are fingering, so to speak, the computer code which is the raw data for the CRU results. That code itself seems deeply problematic, which is not so much a measure of wilful malfeasance as of translating vast amounts of uneven and varied historical data into readable consistent form by scientists already looking for a pattern. In that process, the temptation to massage what is unmassageable may be real - and human (as scientists, for all their pretensions, are). A reader writes:

The really interesting story in Climategate is in the computer code and related files from CRU.  See HARRY_READ_ME.txt online at  What the code and the Read_me file tells us:

(1) CRU's data files of weather-related information gathered at worldwide sites were (are?) a hopeless mess. 

(2) The programmer applied arbitrary adjustments to the data (he says so himself) to get the desired results.

Perhaps this is an isolated case of incompetence, and other researchers have good data and programs.  Perhaps the research created at CRU was somehow good research, despite appearances.  Or perhaps at least some of these scientists are fooling themselves.

Richard Feynman, the brilliant physicist, in his commencement address (PDF) at CalTech related a story of a famous scientist whose published result turned out to be a little bit off.

As he says,

"It's interesting to look at the history of measurements of the charge of the electron, after Millikan.  If you plot them as a function of time, you find that one is a little bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher.  Why didn't they discover that the new number was higher right away?  It's a thing scientists are ashamed of--this history--because it's apparent that people did things like this:  When they got a number that was too high above Millikan's they thought something must be wrong--and they would look for and find a reason why something might be wrong.  When they got a number closer to Millikan's value they didn't look so hard.  And so they eliminated the numbers that were too far off, and did other things like that."  

Feynman knew that learning not to fool yourself was one of the hardest parts of becoming a scientist. 

Scientists are as prone as anyone to taking part in "informational cascades," particularly when they are being funded by granting agencies that reward those who continue in an established line of inquiry (can you imagine funding going to a scientist who found a climate counter-trend?) and when they are trying to publish in peer-reviewed journals whose editorial staff refuse to consider papers that do not come up with the expected results.

Jump to comments

2006-2011 archives for The Daily Dish, featuring Andrew Sullivan

Get Today's Top Stories in Your Inbox (preview)

CrossFit Versus Yoga: Choose a Side

How a workout becomes a social identity

Join the Discussion

After you comment, click Post. If you’re not already logged in you will be asked to log in or register. blog comments powered by Disqus


CrossFit Versus Yoga: Choose a Side

How a workout becomes a social identity


Is Technology Making Us Better Storytellers?

The minds behind House of Cards and The Moth weigh in.


A Short Film That Skewers Hollywood

A studio executive concocts an animated blockbuster. Who cares about the story?


In Online Dating, Everyone's a Little Bit Racist

The co-founder of OKCupid shares findings from his analysis of millions of users' data.


What Is a Sandwich?

We're overthinking sandwiches, so you don't have to.


Let's Talk About Not Smoking

Why does smoking maintain its allure? James Hamblin seeks the wisdom of a cool person.