Employment Exam Writers Tread Carefully
Recent Discrimination Ruling Renews Commitment to Creating Fair Tests

By Krissah Thompson
Washington Post Staff Writer
Monday, September 7, 2009

It is a simple question, pulled straight from an employment test: "When firefighters arrive on the scene of a house fire, what should the lieutenant do first?" Simple enough, but wrapped into the answer is the complicated issue of employment discrimination.

For a few weeks early this summer, a noisy argument about such test questions took over Washington. The Supreme Court ruled that New Haven, Conn., had gone too far by tossing out a test for firefighters because only white applicants scored high enough to get the jobs. Two months later, the passionate national debate over the ruling in Ricci v. DeStefano may have faded, but in the small slice of the economy occupied by the people who write employment exams for a living, racial differences in testing are still the hot topic. The test-makers are trying to figure out how to adjust to the ruling, consumed with writing fair tests that avoid legal challenges.

At the heart of the matter is this: Year after year, test scores line up in the same race-regimented order, with whites and Asians faring better on most standard employment exams than blacks and Latinos.

Many of the people who grapple with the issue live in the Washington region, which is home to one of the largest concentrations of test writers in the country. They are PhDs employed by federal agencies, government contractors, local governments and private companies to select new employees and figure out who moves up the ladder -- and all have different approaches.

Among the most unorthodox is one that has been developed by James Outtz, an independent consultant in downtown Washington, who believes it is possible to do an end run around the testing status quo.

"What I swim upstream against every single day is people saying that if you do something different, you are lowering the standards simply because you did something different, assuming . . . it isn't as good as what you did before," Outtz said.

One test written by Outtz, which he said is "not your typical multiple-choice exam," has been used by a local utility to promote managers. It begins with a video where actors role-play an office conflict. During the exercise, the test takers watch a conflict play out between Ray and Jane. Jane accuses Ray of manipulating their company's projection figures. The test taker must decide how to respond and is given answers to choose from based on the video. Some questions have as many as 26 choices. Why so many?

"Multiple choice testing, in addition to whatever else you are trying to measure, measures something called convergent thinking. You are given a problem and you are asked to converge on a single answer that solves that problem. In real life, we don't have many problems like that," Outtz said. "When I was growing up as a boy and I was poor, I didn't find one way to do anything."

He thinks that in some cultures divergent thinking is valued over convergent thinking. Rather than looking for one answer, he said, divergent thinking values finding as many answers as possible. So in the manager's exam designed by Outtz, the test takers can choose as many answers as they wish, but pick the wrong one and it means disqualification or lost points.

'A Lot Better Than None'

Outtz, who co-authored one of the friend-of-the-court briefs that justices read before deciding Ricci, explained his approach recently at a gathering of local test writers at George Mason University's satellite campus. The lights were dim, but 40 sets of eyes were wide and interested as he clicked through a presentation that only another tester could love.

Nearing the end of his presentation, Outtz pulled up slide 47 -- a photo of a two-story home with flames shooting out of the roof. "When firefighters arrive on the scene of a house fire, what should the lieutenant do first?" he asked. The question is part of an exam he gave to a fire department in the Northeast, and the 15 high scorers on the standard multiple choice test were white, Outtz told his colleagues. When he took into account scores on the oral exam that included the photo of the burning house, the 15 eligible for promotions included two minorities.

"That's a lot better than none," Outtz said.

Rich Cober, who is president of the Personnel Testing Council of Metropolitan Washington, was sitting in the audience. Outtz has "a unique perspective -- an expert one, and not one that everyone will agree with," said Cober, who had invited Outtz to speak.

Outtz advocates experimental, creative test writing, but Cober said the renewed debate about racial implications in testing has reminded him of the importance of "sticking to testing fundamentals."

"The important thing is that any organization have a plan," he said.

Documenting Every Step

In some ways, that plan is about a measure known in the industry as the "80 percent rule." U.S. employment law loosely uses that figure as a ruler for gauging hiring discrimination. If a company hires 50 percent of its white applicants but fewer than 40 percent of its black applicants, it should be able to explain the disparity and make the case that its tests are not unfairly weeding out minorities.

Cober's approach is to document every step of the testing process, using computer software to track applicants and their scores. That information is then plugged into an algorithm to determine where an exam stacks up against the 80 percent rule. He uses standard multiple choice and writing tests if he thinks they are the best way to test for a job, but said he also considers the methods Outtz favors. Companies like using tests that simulate real life, but "there is desire, and then there is time and cost," Cober said.

Also in the audience was Eric Dunleavy, an employment psychologist at DCI Consulting Group, a human resources firm focused on race and employment issues. Among the 40 test writers watching the presentation, there were probably 40 approaches to dealing with race, Dunleavy said, which is why, even with Ricci v. DeStefano on the books, the best way to approach race and testing comes down to this:

"I don't think there is a right answer," he said.

View all comments that have been posted about this article.

© 2009 The Washington Post Company