Yet audit and other counter-simulation systems are
typically treated as afterthoughts in the design of incentive
management systems. The school accountability movement is a good
example here. There are many ways of cheating on standardized tests
other than doctoring the answer keys or even using questions from the
test in class exercises. Simulation strategies come in a wide range of
subtleties, and no doubt all of them are being used.
we're literally training children to answer examinations, all school
tests are merely proxies for things we really care about. It isn't hard
to find ways of producing proxy results instead of real ones, for
example by drilling students in four-term verbal analogies [Apple is to
pear is catfish is to: 1) cat 2) salmon 3) fish 4) seafood 5) none of
the above.] The ability to solve such puzzles quickly (and not too
quirkily) isn't a bad proxy measure for a certain kind of reasoning and
interpretive skill, but it's hardly valuable enough to rate the hour a
week it took out of my 11th-grade English class. The goal back then was
to fool the SAT test to get students into good colleges, rather than
fooling the state to get raises for teachers, but the strategy was the
Test results at the level of the school can also be
influenced by managing the population tested; if all the worst students
transfer to other schools, the average score will surely go up. For
better or worse, expulsion has been made difficult, but there are other
ways - incentive-based ways, many of them - to induce the weaker
players to leave the game.
At the other extreme of subtlety,
dropping art and music (or even reducing hours spent on science and
history) and substituting more hours of reading can be thought of
either as cheating (by degrading a set of valued characteristics that
the tests don't happen to measure) or as simply responding as intended
to an incentive system designed to produce literacy at virtually all
costs. In such cases, discussion of the simulation risks and
counter-simulation strategies will require a discussion of just what it
is that the incentive system is trying to produce, and therefore what
it is that the tests are intended to measure.
counter-simulation measures to back up testing, are, of course,
overhead costs of education, as opposed to direct instructional costs.
Many of those most enthusiastic about testing as a management tool in
public education also advocate reducing spending on overhead items, and
in particular central administration, to concentrate resources on
direct instruction. Those two positions may not be in contradiction—there are, obviously, other categories of overhead—but they
certainly are in tension.
My point is not that high-stakes
standardized testing for educational management is good or bad, but
rather that and discussion of that issue without a parallel discussion
of simulation strategies and counter-simulation strategies is
hopelessly inadequate. At minimum, each proposed measure has to pass
the benefit-cost test of being worth more than the resources required
to create and maintain a monitoring system good enough to keep
result-simulation down to an acceptable level.