The king and queen of data-driven education reform — Education Secretary Arne Duncan and former D.C. schools chancellor Michelle Rhee — are starring at the National Data Summit that starts on Wednesday in the nation’s capital.

Amassing and using data to assess schools and principals and students and teachers and school districts and states is, of course, at the heart of the education accountability movement championed by Duncan and Rhee.

For the past decade, with No Child Left Behind and Race to the Top driving education policy, schools have collected oceans of data that were supposed to inform decisions on how best to boost student achievement and close the education gap. That approach didn’t work as advertised — and in fact has left us with an educational mess — but that won’t stop the national obsession with gathering ever more data.

Duncan and Rhee add star power to the summit, where participants will discuss how to transform “education into a data-driven sector” — as if it weren’t already.

But their own “data records,” if you will, haven’t been without problems.

Duncan famously told Congress last March that 82 percent of public schools could be at risk of failing to meet the adequate yearly requirements of No Child Left Behind in 2011. He was accused — by both supporters and critics of NCLB — of hyping the figure to persuade Congress to rewrite No Child Left Behind in the way he wanted. It turned out that Duncan’s prediction was 34 percentage points too high.

Rhee became chancellor of D.C. schools in 2007 vowing to improve schools with a data-driven reform agenda, and even after she left the job in October 2010 her successor carried on with that approach. But she had her problems with data too.

For one thing Rhee’s IMPACT teacher evaluation system, when first introduced, was seeking so much data that it was practically impossible to carry out. It called for teachers to be evaluated by administrators or master teachers on 22 different teaching elements in a 30-minute period five times a year. The 22 elements were eventually knocked down to nine. Some teachers are also evaluated on student standardized test scores. There are now several investigations into possible cheating on standardized testing in D.C. schools when Rhee was chancellor.

There was also a big controversy last year about data she cited regarding her students’ standardized test scores when she was a teacher in Baltimore. Her resume had said, “Over a two-year period, moved students scoring on average at the 13th percentile on national standardized tests to 90 percent of students scoring at the 90th percentile or higher.” It sounds good, but it turns out that those data points were exaggerated.

It’s a pretty good bet nobody is going to bring up these data stumbles at the summit.

The data summit is part of the Data Quality Campaign, which is a national effort by dozens of organizations and funded by grants and contributions from a variety of foundations including the Bill & Melinda Gates Foundation, the Michael & Susan Dell Foundation, the Lumina Foundation for Education, AT&T, and the Birth to Five Policy Alliance.

The campaign, the website says, works to “encourage and support state policymakers to improve the availability and use of high-quality education data to improve student achievement.”

There’s nothing wrong and there can be a lot right with using high-quality education data to improve achievement, of course, but data can never be the whole story. Ensuring that data is high quality, knowing how to use it — and understanding its limitations — is still not the science. A lot of the data we have is junk, but we use it to inform important decisions anyway.

A book entitledThe Sum of Our Discontent: Why Numbers Make Us Irrational, ” written by David Boyle and published in 2001, makes an excellent case that humans have been trying to use data to solve problems for a very long time, and, given the limitations of data, it only works sometimes.

Last October, California Gov. Jerry Brown issued an indictment of data-based school reform in a message he wrote vetoing a bill that would have changed the state’s accountability system for public schools.

He hit the nail on the head. Here’s part of the message:

“Over the last 50 years, academic ‘experts’ have subjected California to unceasing pedagogical change and experimentation. The current fashion is to collect endless quantitative data to populate ever-changing indicators of performance to distinguish the educational ‘good”’from the education “bad.” Instead of recognizing that perhaps we have reached testing nirvana, editorialists and academics alike call for ever more measurement “visions and revisions.”

“A sign hung in Albert Einstein’s office read“Not everything that counts can be counted and not everything that can be counted counts.”

SB547 nowhere mentions good character or love of learning. It does allude to student excitement and creativity, but does not take these qualities seriously because they can’t be placed in a data stream. Lost in the bill’s turgid mandates is any recognition that quality is fundamentally different from quantity.”

It turns out that Einstein didn’t have a sign with that quote on it in his office, and he probably didn’t even say it. But that doesn’t take away from the truth of the sentiment. As the summiteers in Washington seek a data-driven education world, it would be wise for them to consider this.


Follow The Answer Sheet every day by bookmarking