The Fact-Check: Why John Merrow’s Misleading DC Rant Ignores Facts, Research and the Full Truth

Merrow was last seen engaged in a public feud with Success Academy founder Eva Moskowitz over his reporting in a PBS segment on the New York City charter school’s disciplinary practices.

In a recent blog post directed at D.C., he argues, “The approach to ‘education reform’ begun by Michelle Rhee in 2007 and continuing under Kaya Henderson to this day is a failure and a fraud. Washington’s students and teachers deserve better.”

It’s difficult, even impossible, to know for sure, but two recent examinations of this question suggest that D.C.’s test score gains were not largely a result of changing student demographics. One comprehensive study found that “overall, even when accounting for changes in student demographics, test scores in the District have improved substantially, especially in math.”

A more recent look at the data reported that only a fraction of the NAEP gains were likely explained by demographic changes: “NAEP results have increased not just across race, but across all subgroups such as gender, disability status, and ELL status.”

More importantly, though, reading the NAEP tea leaves to try to determine the efficacy of certain policies is entirely misguided, referred to by some as “misNAEPery” — in which sweeping declarations are made without being able to tie the test scores back to specific policies.

To be clear: Merrow is not the only one to do this. Many supporters of D.C. reforms have engaged in this unfortunate practice. But that is no justification for making unsupported claims about D.C. NAEP scores, as Merrow has done.

Like most places that have administered new Common Core-aligned tests, D.C. saw proficiency rates drop — not because students are learning less, but because the bar for proficiency was made more difficult. Since the test is new, there’s no baseline for comparison.

Merrow is certainly entitled to this view, but the empirical research on No Child Left Behind shows that it improved student achievement in math. As Tom Dee, a Stanford professor who has studied the law, told me a while back, “The available research evidence suggests it led to meaningful — but not transformational — changes in school performance.”

The law also produced many unintended consequences, such as teaching to the testing, which may be why Merrow has deemed it a failure. But he ought to acknowledge the real academic gains that NCLB produced as well.

A National Research Council report states that “from what we could determine … the alleged [cheating] violations were likely not widespread enough to have affected citywide scoring levels” and that “new test security measures that have been introduced in D.C. schools.”

I would start with the city’s teacher evaluation program, an aggressive system that has led to performance bonuses and firings. Research on the early implementation of the policy shows it pushed many teachers to improve and led less-effective educators to leave the classroom. On the other hand, there is concerning evidence that the evaluation system unfairly penalizes teachers with the hardest-to-serve kids. High-poverty schools in DC experience significant teacher turnover (around 40 percent in the 2010–11 school year), which, although not necessarily bad, does on average reduce student achievement.

Another study found that the city’s policy of replacing principals deemed ineffective boosted test scores.

On school choice, D.C. charters appear to be relatively successful at improving student achievement.

D.C.’s aggressive school closures produced temporary declines in achievement for students directly affected, though they rebounded fairly quickly. There is no evidence on whether students who would have attended closed schools benefited, though in New York City they did.

However, the number of African-American teachers in D.C. has plummeted in recent years, potentially because of closure and teacher evaluation policies. Research has found that students of color benefit from teacher diversity.

The National Research Council report raised important concerns about how D.C. collects, disseminates, and uses data. It’s also worth noting that enrollment in D.C. Public Schools has increased in recent years, perhaps a sign that families are more confident in the quality of that schools, though we can’t assume cause and effect.

(Disclosures: I worked as a summer intern at StudentsFirst, an organization founded by Michelle Rhee, a few years ago. The Seventy Four’s Editor-in-Chief Campbell Brown sits on Success Academy’s board of directors.)

---

1. As his source, Merrow says, “Earlier this year a report by the National Research Council pointed out that most of the academic gain was likely the result of more affluent families moving into Washington and enrolling their children in public schools.” I took a look at this report and I’m not sure what Merrow is referring to. The study concludes: “The percentage of all students scoring proficient or above in reading and math on the DC CAS increased between 2007 and 2014. The increase is larger for math than it is for reading. The positive trends are also apparent on NAEP.” However the report warns, “Although we can document some of the changes that occurred over the past 7 years, we cannot determine the independent effects of [DC reform efforts] on achievement and attainment. Changes in the demographic composition of D.C.’s public school students, the growth of the charter sector, differences in the programmatic choices made in DCPS and the individual charter schools, and many other changes that have occurred are intertwined with the changes brought by [DC education policies] …. The signs of improvement are positive, but a more complete picture of student outcomes is needed.“ In other words, as far as I can tell, the National Research Council suggests that demographic changes may be an important factor, but certainly does not definitively claim that they led to the observed test score gains. (Return to story)