Start at the most foundational level: What is
causality? I have an engineer's
perspective on this. What I care about
is my ability to predict the effect of my interventions better than I can
without the model.
Consider two questions:
1. - Does A cause B?
2. - If I take action A, will it cause outcome B?
I don't care about the first, or more precisely, I might
care about it, but only as scaffolding that might ultimately help me to answer
For example, in your shoes story, I don't care whether the
characteristic of discomfort cause shoes to be considered attractive. I care about whether, for example, if I take
an existing type of shoes and narrow the toes, this will cause them to get more
coverage in fashion magazines, sell more units or whatever.
In general, the best way to determine this is to take some comfortable
shoes, narrow the toes, and then see what happens to sales. That is, to run an experiment.
There are big problems with this approach. One obvious one is that it is often
impossible or impractical to run the experiment. But even if we assume that I have done
exactly this experiment, I still have the problem of measuring the causal
effect of the intervention. In a
complicated system, like shoe stores, I have to answer the question of how many
pairs I would have sold in the, say, three months after changing my design to
narrow toes - I can't just assume that I would have sold the same number of
wide-toed shoes that I did in the prior three months. For reasons well-known to you, and that I go through
at length in the book, the best way to measure this in a complicated system is
a randomized field trial (RFT) in which I randomly assign some stores to get
the new shoes and others to keep selling the old shoes. In essence, random assignment allows me to
roughly hold constant all of the "screwy" effects that you reference between the
test and control group.
But what many cheerleaders for randomized experiments gloss
over is that even if I have executed a competent experiment, it is not obvious
how I turn this result in to a prediction rule for the future (the problem of
generalization or external validity). Here's
how I put this in an article
a couple of years ago:
In medicine, for example, what we
really know from a given clinical trial is that this particular list of
patients who received this exact treatment delivered in these specific clinics
on these dates by these doctors had these outcomes, as compared with a specific
control group. But when we want to use the trial's results to guide future
action, we must generalize them into a reliable predictive rule for
as-yet-unseen situations. Even if the experiment was correctly executed, how do
we know that our generalization is correct?
A physicist generally answers that
question by assuming that predictive rules like the law of gravity apply
everywhere, even in regions of the universe that have not been subject to
experiments, and that gravity will not suddenly stop operating one second from
now. No matter how many experiments we run, we can never escape the need for
such assumptions. Even in classical therapeutic experiments, the assumption of
uniform biological response is often a tolerable approximation that permits
researchers to assert, say, that the polio vaccine that worked for a test
population will also work for human beings beyond the test population.