This ability to rapidly test ideas fundamentally changes the company's mindset and approach to innovation. Rather than agonize for months over a choice, or model hypothetical scenarios, the company simply asks the customers and get an answer in real time.
According to Google economist Hal Varian, his company is running on the order of 100-200 experiments on any given day, as they test new products and services, new algorithms and alternative designs. An iterative review process aggregates findings and frequently leads to further rounds of more targeted experimentation.
At the same time, Google's competitors, partners, customers and third party consultants are doing their own experiments, creating a complex, interacting ecosystem that demands continuous innovation. While Google currently dominates the market for web search, it is unlikely that it would have any market share at all if it still relied on the original, unmodified PageRank algorithm that Larry Page and Sergey Brin developed in 1998.
HOW DATA TURNED AROUND A CASINO
Greg Linden, who led one set of experiments at Amazon, describes the emerging experimentation philosophy succinctly: "To find high impact experiments, you need to try a lot of things. Genius is born from a thousand failures. In each failed test, you learn something that helps you find something that will work. Constant, continuous, ubiquitous experimentation is the most important thing."
These words echo the approach of innovators since Thomas Edison, but IT has made it possible to apply it to a much broader class of business challenges and significantly compress the "hypothesis-to-experiment" cycle time.
While web-based companies have been particularly aggressive in using business experiments to drive innovation, other industries are getting in the game. Caesar's Entertainment (formerly Harrah's), the hotel and casino company, transformed itself from a 2nd-tier casino to an industry leader in large part because of the culture of experimentation introduced by CEO Gary Loveman.
When Loveman, an economics PhD from MIT and former Harvard Business School professor, arrived at the company, he found that it was already gathering a great deal of data about its customer interactions with existing information systems and programs such as its Total Rewards loyalty card. However, it wasn't using these data to develop improved processes, products and services. After becoming CEO, he developed strategies to continually tests new promotions, price points, services, workflow, employee incentive plans and casino layouts using controlled experiments.
Widespread business experimentation has required a fundamental change in the corporate culture. As Loveman puts it "There are two things that will get you fired here: stealing from the company, or running an experiment without a properly designed control group."
While passive data gathering can be useful, measurement is far more valuable when coupled with conscious, active experimentation and sharing of insights. Likewise, the value of undertaking the experiments themselves is proportionately greater if the organization can capitalize on those experiments in more locations and at greater scale. In combination, these practices constitute a new kind of "R&D" that draws on the strengths of digitization to speed innovation.