Education has entered the era of Big Data. The Internet is teeming with stories touting the latest groundbreaking studies on the science of learning and pedagogy. Education journalists are in a race to report these findings as they search for the magic formula that will save America's schools. But while most of this research is methodologically solid, not all of it is ready for immediate deployment in the classroom.
Jessica was reminded of this last week, after she tweeted out an interesting study on math education. Or, rather, she tweeted out what looked like an interesting study on math education, based on an abstract that someone else had tweeted out. Within minutes, dozens of critical response tweets poured in from math educators. She spent the next hour debating the merits of the study with an elementary math specialist, a fourth grade math teacher, and a university professor of math education.
Tracy Zager, the math specialist, and the author of the forthcoming book Becoming the Math Teacher You Wish You’d Had, emailed her concerns about the indiscriminate use of education studies as gospel:
Public education has always been politicized, but we've recently jumped the shark. Catchy articles about education circulate widely, for understandable reason, but I wish education reporters would resist the impulse to over-generalize or sensationalize research findings.
While she conceded that education journalists “can’t be expected to be experts in mathematics education, or science education, or literacy education,” she emphasized that they should be held to a higher standard than the average reader. In order to do their jobs well, they should not only be able to read studies intelligently,“they should also consult sources with field-specific expertise for deeper understanding of the fields.”
After she was schooled on Twitter, Jessica called up Ashley Merryman, the author of Nurture Shock: New Thinking About Children, and Top Dog: The Science of Winning and Losing. “Just because something is statistically significant does not mean it is meaningfully significant,” Merryman explained. “The big-picture problem with citing the latest research as a quick fix is that education is not an easy ship to turn around.” When journalists cite a press release describing a study without reading and exploring the study’s critical details, they often end up oversimplifying or overstating the results. Their coverage of education research therefore could inspire parents and policymakers to bring half-formed ideas into classroom. Once that happens, said Merryman, “the time, money, and investment that has gone into that change means we are stuck with it, even if it’s later proven to be ineffective in practice.”
As readers and writers look for solutions to educational woes, here are some questions that can help lead to more informed decisions.
1. Does the study prove the right point?
It’s remarkable how often far-reaching education policy is shaped by studies that don’t really prove the benefit of the policy being implemented. The Tennessee Student Teacher Achievement Ratio (STAR) study is a great example.
In the late 1980s, researchers assigned thousands of Tennessee children in grades K-3 to either standard-sized classes (with teacher-student ratios of 22-to-1) or smaller classes (15-to-1) in the same school and then followed their reading and math performance over time. The landmark STAR study concluded that K-3 kids in smaller classes outperformed peers in larger classes. This led to massive nationwide efforts to achieve smaller class sizes.
Subsequent investigations into optimal class size have yielded more mixed findings, suggesting that the story told in STAR was not the whole story. As it turns out, the math and reading benefits experienced by the K-3 kids in Tennessee might not translate to eighth grade writing students in Georgia, or geography students in Manhattan, or to classes taught using different educational approaches or by differently skilled teachers. A key step in interpreting a new study is to avoid extrapolating too much from a single study, even a well-conducted one like STAR.