The data revolution has turned customers into unwitting business consultants, as our purchases and searches are
tracked to improve everything from websites to delivery routes
In the 1670s, in Delft, Netherlands, a scientist named Anton van Leeuwenhoek did something many scientists had done for 100 years before him. He built a microscope.
This microscope was different, but it was not extraordinary. Like so many inventions, he borrowed and tweaked his predecessors' ingenuity. But when he looked through this microscope, he found things that did seem extraordinary. He called them "animalcules," microbes in water droplets and human blood that ultimately provided the foundation for the germ theory of disease and eventually inspired a host of medicines and treatments.
The Leeuwenhoek discovery is crucial to our understanding of innovation, not only because it changed the face of biochemistry, but also because it represents a fundamental theme of discovery.
Breakthroughs in innovation often rely on breakthroughs in measurement.
THE DATA BOOM
Today businesses can measure their activities and customer relationships with unprecedented precision. As a result, they are awash with data. This is particularly evident in the digital economy, where clickstream data give precisely targeted and real-time insights into consumer behavior.
In turn, customers are acting as unwitting business consultants for these companies. Our purchases, searches, and online activities are being tracked to improve everything from websites to delivery routes and drug manufacturing.
Anyone with access to a Web browser can get summaries of billions of keyword searches, and this information is highly predictive of present and future economic activity, such as housing purchases and prices. Mobile phones, automobiles, factory automation systems and other devices are routinely instrumented to generate streams of data on their activities, making possible an emerging field of "reality mining" to analyze this information. Manufacturers and retailers use radio-frequency identification (RFID) tags to deliver terabits of data on inventories and supplier interactions and then feed this information into analytical models to optimize and reinvent their business processes.
Much of this information is generated for free, by computers, and sits unused, at least initially. A few years after installing a large enterprise resource planning system, it is common for companies to purchase a "business intelligence" module to try to make use of the flood of data that they now have on their operations. As Ron Kohavi at Microsoft memorably put it, objective, fine-grained data are replacing HiPPOs (Highest Paid Person's Opinions) as the basis for decision-making at more and more companies. For example:
-- Enologix has used this approach to help Gallo vineyards accurately predict the wine ratings that Robert Parker would give to various new wines
-- UPS has mined data on truck delivery times to develop a new routing method
-- Match.com as even developed new algorithms for matching men and women for dates
For each innovation, analysts drew on new measurement technologies to supplant human experts who relied more on intuition. However, for all its strengths, measurements have a shortcoming. They cannot determine causality. (A simple example: Shoe sizes and readings scores are correlated for school children, but one does not cause the other; instead, they both reflect a third variable, which is age.) Fortunately, science has a second powerful tool designed precisely to address questions of causality.
That tool is called experimentation.
AN EXPERIMENT EVERY SECOND
Science has been dominated by the experimental approach for nearly 400 years. Running controlled experiments is the gold standard for sorting out cause and effect. But experimentation has been difficult for businesses throughout history because of cost, speed and convenience. It is only recently that businesses have learned to run real-time experiments on their customers. The key enabler was the Web.