Employment Exam Writers Tread Carefully
Monday, September 7, 2009
It is a simple question, pulled straight from an employment test: "When firefighters arrive on the scene of a house fire, what should the lieutenant do first?" Simple enough, but wrapped into the answer is the complicated issue of employment discrimination.
For a few weeks early this summer, a noisy argument about such test questions took over Washington. The Supreme Court ruled that New Haven, Conn., had gone too far by tossing out a test for firefighters because only white applicants scored high enough to get the jobs. Two months later, the passionate national debate over the ruling in Ricci v. DeStefano may have faded, but in the small slice of the economy occupied by the people who write employment exams for a living, racial differences in testing are still the hot topic. The test-makers are trying to figure out how to adjust to the ruling, consumed with writing fair tests that avoid legal challenges.
At the heart of the matter is this: Year after year, test scores line up in the same race-regimented order, with whites and Asians faring better on most standard employment exams than blacks and Latinos.
Many of the people who grapple with the issue live in the Washington region, which is home to one of the largest concentrations of test writers in the country. They are PhDs employed by federal agencies, government contractors, local governments and private companies to select new employees and figure out who moves up the ladder -- and all have different approaches.
Among the most unorthodox is one that has been developed by James Outtz, an independent consultant in downtown Washington, who believes it is possible to do an end run around the testing status quo.
"What I swim upstream against every single day is people saying that if you do something different, you are lowering the standards simply because you did something different, assuming . . . it isn't as good as what you did before," Outtz said.
One test written by Outtz, which he said is "not your typical multiple-choice exam," has been used by a local utility to promote managers. It begins with a video where actors role-play an office conflict. During the exercise, the test takers watch a conflict play out between Ray and Jane. Jane accuses Ray of manipulating their company's projection figures. The test taker must decide how to respond and is given answers to choose from based on the video. Some questions have as many as 26 choices. Why so many?
"Multiple choice testing, in addition to whatever else you are trying to measure, measures something called convergent thinking. You are given a problem and you are asked to converge on a single answer that solves that problem. In real life, we don't have many problems like that," Outtz said. "When I was growing up as a boy and I was poor, I didn't find one way to do anything."
He thinks that in some cultures divergent thinking is valued over convergent thinking. Rather than looking for one answer, he said, divergent thinking values finding as many answers as possible. So in the manager's exam designed by Outtz, the test takers can choose as many answers as they wish, but pick the wrong one and it means disqualification or lost points.
'A Lot Better Than None'
Outtz, who co-authored one of the friend-of-the-court briefs that justices read before deciding Ricci, explained his approach recently at a gathering of local test writers at George Mason University's satellite campus. The lights were dim, but 40 sets of eyes were wide and interested as he clicked through a presentation that only another tester could love.
Nearing the end of his presentation, Outtz pulled up slide 47 -- a photo of a two-story home with flames shooting out of the roof. "When firefighters arrive on the scene of a house fire, what should the lieutenant do first?" he asked. The question is part of an exam he gave to a fire department in the Northeast, and the 15 high scorers on the standard multiple choice test were white, Outtz told his colleagues. When he took into account scores on the oral exam that included the photo of the burning house, the 15 eligible for promotions included two minorities.
"That's a lot better than none," Outtz said.