Perhaps the strangest thing about the heated debate over the Common Core standards for math is the fact that we are debating them at all. The standards themselves tend to be dry and abstract and have no direct impact on students. This is not to say that they are not important, but that their importance comes through the effect they have on curriculum and testing. All of which suggests that we should perhaps be spending less time discussing the contents of the standards and more time discussing their implementation.

In terms of the materials that students actually use in the classroom and as homework, the acknowledged leader at implementing the math standards is EngageNY’s Eureka Math.

Rachel Monahan writes:

Louisiana published a review of Common Core curricula last year, and gave EngageNY’s Eureka Math a top ranking, leading a large number of districts in the state to adopt it, according to officials with the nonprofit Great Minds Inc., which developed Eureka Math for EngageNY. Parts of Core Knowledge also ranked at the top along with one textbook series.

Favorable reviews there and elsewhere of the free EngageNY materials have helped expand interest. Achieve, a nonprofit group that backs the Common Core standards, gave three Expeditionary Learning units its highest ranking.

“Do we want as many districts as possible adopting our stuff? Sure. Am I more interested in our setting a new standard for the quality of instructional materials in reading and math? Yes,” said Great Minds executive director Lynne Munson. “I’m vastly more interested in seeing that more students are learning these subjects well than in selling anything to anyone.”

Of course, we can all agree on the importance of quality; the trouble comes when we start trying to define it. The reviews mentioned above are focused on alignment with the Common Core standards, not on the materials being accurate or clear or interesting. That focus doesn’t make the reviews wrong, but it can create a major disconnect in perceived quality, with government officials and policy experts on one side and kids, parents and teachers on the other.

How large is this disconnect? Recently blogger and veteran math teacher Gary Rubinstein did an informal spot check of Eureka’s materials and was appalled by what he found:

First of all, some lessons are full of errors. Second, some lessons are unnecessarily boring, and third, some lessons are unnecessarily confusing.

Those mistakes Rubinstein found were not just frequent; they were big:

Exhibit A is the first lesson in the first module for 8th grade, exponents.  On the second page, they introduce the concept of raising a negative number to a positive integer.  Every real math teacher knows that there is a difference between the two expressions (-2)^4 and -2^4.  The first one means (-2)*(-2)*(-2)*(-2)=+16 while the second one, without the parentheses around the -2 means -1*2*2*2*2=-16.  I have checked with all the math teachers I know, and none have ever seen -2^4 interpreted as (-2)^4.  Yet, here all over lesson one module one for 8th grade EngageNY teacher’s edition, we see this mistake.

screen-shot-2015-05-17-at-11-22-57-am1

What is worse, Rubinstein found this same mistake 18  of the 20 times the topic was covered.

After reading this, I went to the EngageNY site to do some checking of my own and found just what Rubinstein had described: errors, boredom and confusion.

When writing high school or junior high math books, there is always a struggle between using formal and precise terms and keeping things understandable. You want to be as accurate as possible but you want to avoid dense, hard-to-follow technical language. The ideal is right but readable.

All too often, the materials from Eureka managed to do just the opposite. They introduced concepts with the kind of prose you would associate with a graduate level mathematics course, then proceeded to get the problem wrong on the most fundamental level.

Case in point, this is how a “tips for parents” lesson explains the concept of similarity (which is, in everyday language, “having the same shape”).

Similarity Transformation: A similarity transformation, or similarity, is a composition of a finite number of basic rigid motions or dilations. The scale factor of a similarity transformation is the product of the scale factors of the dilations in the composition; if there are no dilations in the composition, the scale factor is defined to be 1.

A little bit later, the authors talk about corresponding sides between a rectangle and a triangle. Not to dredge up too many memories of school geometry, but the concept of corresponding sides only applies to polygons that have the same number of sides. After dragging students through the most technical of definitions, the authors get the most basic of terms wrong.

That combination of dense and formal language followed by incorrect problems is even more noticeable in the statistics sections. For example, in the lesson on interpreting research studies in the Algebra II section, we get the following paragraph:

Students should look to see if the article explicitly states that the subjects were randomly assigned to each treatment group. This is important because random assignment negates the effects of extraneous variables that may have an effect on the response by evenly distributing these variables into both treatment groups.

It is difficult to imagine the typical algebra student getting much out of the phrase “negates the effects of extraneous variables that may have an effect.” This paragraph needs a much more readable explanation (and probably one fewer “effect”). Then the authors proceed to confuse causal and deterministic relationships in a way that is guaranteed to send researchers and statisticians screaming from the room.

It is difficult to say how big the issues with Eureka math are. All of preceding examples are based on a few informal spot checks of the junior high and high school curriculum. We have no way of estimating how widespread these problems are in these grades, nor can we make inferences about the state of the lower grades, but even if this sample should prove highly unrepresentative, the concerns cited here are still serious enough to reveal a serious quality issue with EngageNY, an organization that is providing instructional materials for millions of children.

Political science is relevant to this discussion. Clueless people designing stupid lessons—that’s no big deal, it happens. But these lessons being considered the state of the art for a national curriculum—it takes politics for that to happen.

Mark Palko is an L.A.-based statistician who blogs at West Coast Stat Views and You Do the Math and is author of Things I Saw at the Counter-Reformation: A look back at the bad statistics, questionable pedagogy, and strange bedfellows of the education reform movement circa 2010.