wpostServer: http://css.washingtonpost.com/wpost

The Post Most: Local

Answer Sheet
Posted at 10:57 PM ET, 07/26/2011

Was a NAEP geography test question poorly constructed?

Last week I wrote about the newly released 2010 geography scores on the National Assessment of Educational Progress (NAEP), which showed that students in fourth, eighth and twelfth grades don’t know a whole lot about the subject.

The post (which was excerpted in The Washington Post on Monday) included an eighth-grade question about the American Southwest that had been on the exam. Now, Candace Matthews, who has worked in the field of education and training for 35 years, has raised concerns about just how well drawn the test question actually was and what this may mean for other questions. Matthews now works for the federal government in the field of training.

Her questions are interesting because NAEP is considered the gold standard of assessment, and is often called America’s report card because it is the only exam that tests nationally representative groups of K-12 students at different grades in a range of subjects.

This is the question that I included in the post and that is posted on the website of the National Assessment Governing Board, the body that oversees the content and administration of the NAEP, along with a release on the scores:

Which of the following is an accurate statement about the American Southwest?

a) Alternating areas of dense shrubbery and sand dunes often make travel difficult.

b) Arid conditions make access to water an important public issue.

c) Generally fair weather means that most people rely on solar energy in their homes and businesses.

d) Easy access to Mexico has led to a strong manufacturing sector.

The correct answer is B, and only 33 percent of eighth-grade students who took the test got it right.

Here’s what Matthews wrote to me:

One point particularly drew my attention in the excerpt from your blog published in the Washington Post (7/25/2011). Rather than being surprised by the lack of student knowledge revealed by the results of the 2010 National Assessment of Educational Progress, I was surprised by the poor quality of the test item you presented from that test. In order to make valid inferences from test results, we need to understand the elements of an effective test item. In this case, I believe that poor construction of the test item should raise some questions.

The principles of constructing effective multiple-choice test items are well documented. And the results of violating these standard item-writing principles are flawed test items that lessen the validity of the test results. The item about the American Southwest clearly violates at least two of these basic principles:

1. Write the stem as a single, clearly formulated problem or question that is possible to answer without reading the options. The stem should be meaningful by itself. This means that test writers should avoid using unfocused stems that require students to read all of the options in order to understand the point of the item. Unfocused items can often be viewed as a set of unrelated or miscellaneous true/false items grouped under a common stem—a format not recommended for high-stakes tests since it is detrimental to item performance.

2. Use precise language, avoiding vague and imprecise terms that are open to personal interpretation.

Here’s the item:

Which of the following is an accurate statement about the American Southwest?

A. Alternating areas of dense shrubbery and sand dunes often make travel difficult.

B. Arid conditions make access to water an important public issue.

C. Generally fair weather means that most people rely on solar energy in their homes and businesses.

D. Easy access to Mexico has led to a strong manufacturing sector.

The stem most definitely does NOT provide a single clearly formulated question or problem. While the item identifies the American Southwest, is it testing student knowledge about topography (option A), transportation issues (option A), climate (options B and C), public priorities (option B), primary energy sources (option C), geography (option D), or the economy (option D). If students select one of the incorrect options, can we infer that they don’t know about the climate of the American Southwest? The answer is no. What if they know that the climate is very dry, but they don’t know what “arid” means? Furthermore, the item seems to add unnecessary confusion by testing cause-effect relationships in each of the options.

Further questions arise if you examine the options more closely.

*In option A, for example, considering the well-developed US highway system, would any type of topography “make travel difficult?” Does this option refer to travel by car, by foot, by train, by plane? [Would the Rocky Mountains “make travel difficult” if one is traveling on a state highway?] I believe the wording of this option is confusingly vague.

*What does “generally fair weather” mean? Here again, I believe the wording is confusingly vague. Does this mean that the sun usually shines in the American Southwest? Does this mean that the weather is generally pleasant? [Both definitions provided by online dictionaries.]

*What does “easy access to Mexico” mean? Do all the states in the American Southwest even have the same “ease of access” — whatever that means — to Mexico? Do people, cities, or businesses in northern Arizona (for example) have “easy access to Mexico”? Does that mean close proximity? Does that mean that it’s easy to cross the border to Mexico? Does that mean it’s easy to move goods between the two countries? Yet again, I believe the wording is confusingly vague.

I am aware that the NAEP is a highly regarded test, but this test item certainly seems flawed. And since this one item is flawed, it raises questions about the quality of the other items and thus the validity of the inferences drawn from the results.

References:

Haladyna, Thomas M. Developing and Validating Multiple-Choice Test Items, 3rd Edition, New Jersey: Lawrence Erlbaum Associated, Publishers, 2004.

Shrock, Sharon & William Coscarelli. Criterion-Referenced Test Development: Technical and Legal Guidelines for Corporate Training and Certification, 2nd Edition, Maryland: International Society for Performance Improvement, 2000.

-0-

Follow The Answer Sheet every day by bookmarking http://www.washingtonpost.com/blogs/answer-sheet. And for admissions advice, college news and links to campus papers, please check out our Higher Education page. Bookmark it!

By  |  10:57 PM ET, 07/26/2011

 
Read what others are saying
     

    © 2011 The Washington Post Company