Why we should track and measure everything

Via Alan Jacobs, I came across a short, interesting post from Doug Bowman, one of Google's lead designers, on why he's leaving the company.

Without a person at (or near) the helm who thoroughly understands the principles and elements of Design, a company eventually runs out of reasons for design decisions. With every new design decision, critics cry foul. Without conviction, doubt creeps in. Instincts fail. "Is this the right move?" When a company is filled with engineers, it turns to engineering to solve problems. Reduce each decision to a simple logic problem. Remove all subjectivity and just look at the data. Data in your favor? Ok, launch it. Data shows negative effects? Back to the drawing board. And that data eventually becomes a crutch for every decision, paralyzing the company and preventing it from making any daring design decisions.

It's hard not to sympathize with Bowman. This reliance on data and experimentation does seem a little soulless. And to Bowman's credit, he acknowledges why Google takes this approach.

I can't fault Google for this reliance on data. And I can't exactly point to financial failure or a shrinking number of users to prove it has done anything wrong. Billions of shareholder dollars are at stake. The company has millions of users around the world to please. That's no easy task. Google has momentum, and its leadership found a path that works very well.

Bowman is not advancing any grand epistemological argument here; rather, he's offering a modest lament concerning the constraints placed on creativity at a multi-billion dollar enterprise. My own view, however, is that we don't rely on data and experimentation enough, and that we ought to embrace what Bowman describes as the engineer's perspective when we consider bold strikes in our lives as individuals and in considering, say, the future fate of government. One of our fellow bloggers, Jim Manzi, has written very persuasively on this subject.

I spent about the first ten years of my career executing increasingly sophisticated quantitative analyses that used data to try to evaluate and predict the success of business initiatives in order to develop corporate strategies. Eventually, I saw that these analyses led to the same kind of scholastic debates as we see among macroeconomists. The root issue was that it was impossible to find a methodology that could reliably distinguish correlation from causality. Only through exhaustion of all possible alternatives did I come to find that experiments that randomly assign units of analysis (customers, stores, sales territories, etc.) to test and control groups are the only reliable method for determining causality.

This is an insight that I'd submit probably extends even to our personal lives, hence the growing popularity of psychologist Seth Roberts brilliant self-experimentation framework, which he chronicles at his blog. Roberts is best known for his "Shangri-La Diet," which he describes as a surefire way to suppress appetite. He discovered it after conducting dozens of uncontrolled experiments on himself, which is also how he cured his insomnia and his acne.

I'm guessing that I'm going further than Jim would, but this is why I'm so excited about Microsoft researcher Gordon Bell's brilliant MyLifeBits project, an effort to construct a truly comprehensive "lifestream" that captures everything Bell reads and sees and writes, and by implication everything he senses and thinks. Imagine if millions gathered similarly comprehensive lifestreams, and if we could compare and contrast different ways of living, ranging from the different ways we tie our shoes to how we balance work and leisure. This rich dataset wouldn't "answer" any questions per se. But it would enrich our perspective, and remind us that we are not alone in the universe -- that we humans are, as Mormon iconography suggests, more like bees in a beehive than solitary animals.

Bowman is a partisan of serendipity -- of intuition and bold strokes. And of course there is always a place for that in life. That said, exposure to a rich library of experience needn't be the enemy of that, not least because tastes aren't fixed. That's why we have tastemakers, to stretch the boundaries of our sensibilities and aesthetic preferences.

To some extent, the consumer marketplace rewards perversity -- products that shock us into living in a new and different way, like the automobile, are the ones that endure. So in that regard I stress that Bowman's view is vitally important.

But can I please have my data? Tim O'Reilly and Joshua Michele-Ross of O'Reilly Radar have been riffing on this theme rather brilliantly.

In the era of the network, the key competency is harnessing collective intelligence. But Josh hammers home the further insight, namely that these effects are not limited to cyberspace, but are used to control and coordinate real-world activity. This is the new frontier, moving from "sensing" to "reacting," from "cognition" to "coordination" and group action. 

That is, how do we use the data that we collect? This is an issue of particular concern to those who want to improve the quality of public policy -- to take society's temperature, so to speak, in real-time.

This leads me, briefly, to another buzzy notion that emerged from Southby this year: the focus on government and education, and transparency in both. More on that to come.