Trying to assess learning gives colleges their own test anxiety

March 14, 2012

Eight years ago, leaders of the University of Texas set out to measure something few in higher education had thought to question — how much their students learn before graduation.

An unsettling answer emerged: arguably, not very much.

That conclusion is based on results from a 90-minute essay test given to freshmen and seniors that aims to gauge gains in critical thinking and communication skills.

The Texas flagship and a few hundred other public universities have joined a growing accountability movement in higher education, embracing this test and others like it that attempt, for the first time, to quantify collegiate learning on a large scale.

But the results have triggered a wave of rancor. Some college leaders are outraged that four years of learning might now be reduced to a single score. Lackluster results have seeded fresh doubts about the country’s vaunted system of higher education.


Left to right, University of Texas at Austin students Jenna Shorter, Nicole Scallan, Charles Gee and Katie Schroeder listen to their professor, Mary Worthy, during a freshman seminar class called The Importance of Interest in Learning and Life. (Nuri Vallbona/For The Washington Post)

“Oh, it’s hit us in the gut,” said Andrew Hacker, a Queens College political scientist and authority on college teaching.

The Collegiate Learning Assessment, launched in 2000, has brought rare scrutiny to higher education. Until now, colleges have been largely exempt from the accountability movement sweeping through public elementary and secondary schools yielding the No Child Left Behind law and other initiatives.

In a landmark study published last year, sociologists Richard Arum and Josipa Roksa used the test to measure collegiate learning in the nation. Using data drawn from a sampling of public and private colleges, they shook the academic world with a finding that 36 percent of students made no significant learning gains from freshman to senior year.

I think it’s extremely troubling,” said Margaret Spellings, the former U.S. education secretary and a longtime advocate of accountability in education. “And God bless Richard Arum for taking this on.”

But a chorus of college leaders reject that this test — or any other — can affirm or refute the essential value of college. They view the CLA as a rough gauge of student learning, grossly inadequate to measure an entire institution.

“I think it’s a very worthwhile attempt,” said William E. “Brit” Kirwan, chancellor of the Maryland state university system. “I don’t think it should be seen as the final answer.”

From time to time, accountability advocates have proposed requiring colleges to show the value they add to the quest for knowledge as a condition of receiving federal aid. But higher education lobbyists and their allies in Congress have “vigorously opposed” attempts to impose a No Child-style system on academia, Spellings said. Perhaps the biggest fear among college presidents is that published test scores might be put to ill use by the collegiate ranking industry.


Brian Hooks, left, and Jessica Lund huddle together as they study in the new Student Activities Center on the University of Texas at Austin campus. (Nuri Vallbona/For The Washington Post)

Yet a voluntary system of accountability is underway. Two groups representing more than 500 public colleges have pledged to give the CLA or one of its rival tests — the Collegiate Assessment of Academic Proficiency or Proficiency Profile — and to publish results by the end of this year. So far, 144 schools have posted test results, including Frostburg State University in western Maryland. Many schools have not participated in the testing, including the universities of Virginia and Maryland.

Teresa Sullivan, president of U-Va., said the University of Michigan gave the CLA when she was provost there. Freshmen scored so high, she said, there was no way for seniors to score higher. She believes U-Va. students would hit the same ceiling.

“If there is no way to improve, why would you invest your money in this?” she said.

The CLA is an essay exam that tests students on skills colleges avow to teach. Responses are judged on use of language, organizational structure and persuasive heft. Students might be asked, for example, to assail the logic in this proposition: Couples should not wed in June, because many failed marriages begin as June weddings.

The University of Texas, one of the nation’s top research universities, was among the first to give the CLA and is using the results to improve instruction. Testing began in 2004 in Austin, under a state mandate.

Last year, UT freshmen scored an average 1261 on the assessment, which is graded on a scale similar to that of the SAT. Seniors averaged 1303. Both groups scored very well, but seniors fared little better than freshmen, according to score reports The Washington Post obtained through a public records request.

“The seniors have spent four years there, and the scores have not gone up that much,” said Arum, a New York University sociologist and co-author of the 2011 book “Academically Adrift.” He reviewed UT’s results at the request of The Post. The school was not among the 24 unnamed colleges in Arum’s study.

With about 51,000 students and nearly 3,000 faculty, the University of Texas is a veritable learning factory. Critics of the CLA, and there are many in Austin, say it is absurd to judge an organization of this scale on the strength of one score. They note that the CLA is relatively brief and administered to a couple hundred freshmen and seniors who have no stake in the results.

University of Texas leaders share Sullivan’s concern that they look bad on the assessment because freshman scores are so high. That is why few highly selective institutions participate in the CLA, university officials say.

The test “is aimed a bit low for the kind of students we get,” said Paul Woodruff, dean of undergraduate studies at UT.

Thor Lund, 20, a senior, is dubious of the value of the CLA. Lund said he scored 34 out of 36 on the ACT college entrance exam. “How much better at critical thinking can you get?” he asked.

But the test’s authors contend smart students at some other colleges show plenty of growth on the CLA over time. For learning gains from freshman to senior year, UT ranked in the 23rd percentile among like institutions. In other words, 77 percent of universities with similar students performed better.

In recent years, UT officials have sought to bolster undergraduate instruction through classes that place a greater emphasis on writing.

Nicole Scallan, 19, is enrolled in one such course. “The Importance of Interest in Learning and Life,” an 18-student seminar, is part of a series of “signature courses” introduced in 2007 and required of freshmen.

One recent morning, instructor Mary Worthy told Scallan and her classmates that they would be sharing drafts of a protest song with each other.

“Did any of you participate in writing workshops when you were in high school or middle school?” Worthy asked, to a collective shaking of heads. “That’s kind of what the idea is.”

By semester’s end, each student will have penned about 75 pages, mostly in two- and three-page assignments. The topics are tailored to engage: the sources of human motivation, the quest for an optimal state of consciousness. Students sit around an oval table and talk, a marked change from sitting in an auditorium and listening.

A 2009 overhaul of the university’s basic education requirements stresses critical thinking and communication skills, qualities measured by the CLA. The UT curriculum lists those skills first among six overarching objectives.

Administrators and faculty infuse courses with opportunities for writing and engagement. The university cannot afford to break up every 400-student lecture into tiny seminars. Instead, professors learn to teach large courses more effectively.

One breakthrough is the electronic “clicker,” which enables professors to pose a question to hundreds of students in real time. Such exercises force students to engage and provides the instructor with a gauge of whether they are learning.

Another innovation is “minimal marking,” an approach to grading that favors broad comments over line editing, a strategy for instructors who assign papers by the hundreds.

“Instead of fixing every comma, you tell the student, ‘You’ve got a comma problem,’ ” said Woodruff, who became UT’s first undergraduate studies dean in 2006.

Gretchen Ritter, vice provost for undergraduate education at UT, has reservations about the CLA. But she concedes the test has inspired the university to question whether students are learning to think.

And that, she said, “is a very good thing to care about.”

SECTION: {section=local, subsection=education}!!!
INITIAL commentConfig: {includereply=true, canvas_permalink_id=washpost.com/8bvh5zpd9k, allow_comments=true, commentmaxlength=2000, includeshare=true, display_comments=true, canvas_permalink_app_instance=m6yzjj840m, display_more=true, moderationrequired=false, includefeaturenotification=true, canvas_allcomments_id=washpost.com/km4ey0dajm, comments_period=14, defaultsort=reverseChronological, includevoteofftopic=false, allow_videos=false, includesorts=true, markerdisplay=post_commenter:Post Commenter|staff:Post Writer|top_commenter:Post Forum|top_local:Washingtologist|top_sports:SuperFan|fact_checker:Fact Checker|post_recommended:Post Recommended|world_watcher:World Watcher|cultuer_connoisseur:Culture Connoisseur|weather_watcher:Capital Weather Watcher|post_contributor:Post Contributor, childrenitemsperpage=3, includeheader=true, includeverifiedcommenters=true, defaulttab=all, includerecommend=true, includereport=true, maxitemstop=3, source=washpost.com, allow_photos=false, maxitems=15, display_ugc_photos=false, includepause=true, canvas_allcomments_app_instance=6634zxcgfd, includepermalink=false}!!!

UGC FROM ARTICLE: {allow_comments=true, allow_photos=false, allow_videos=false, comments_period=14, comments_source=washpost.com, default_sort=, default_tab=, display_comments=true, is_ugc_gallery=false, max_items_to_display=15, max_items_to_display_top=3, moderation_required=false, stream_id=}!!!

FINAL commentConfig: {includereply=true, canvas_permalink_id=washpost.com/8bvh5zpd9k, allow_comments=true, commentmaxlength=2000, includeshare=true, display_comments=true, canvas_permalink_app_instance=m6yzjj840m, display_more=true, moderationrequired=false, includefeaturenotification=true, canvas_allcomments_id=washpost.com/km4ey0dajm, comments_period=14, defaultsort=reverseChronological, includevoteofftopic=false, allow_videos=false, includesorts=true, markerdisplay=post_commenter:Post Commenter|staff:Post Writer|top_commenter:Post Forum|top_local:Washingtologist|top_sports:SuperFan|fact_checker:Fact Checker|post_recommended:Post Recommended|world_watcher:World Watcher|cultuer_connoisseur:Culture Connoisseur|weather_watcher:Capital Weather Watcher|post_contributor:Post Contributor, childrenitemsperpage=3, includeheader=true, includeverifiedcommenters=true, defaulttab=all, includerecommend=true, includereport=true, maxitemstop=3, source=washpost.com, allow_photos=false, maxitems=15, display_ugc_photos=false, includepause=true, canvas_allcomments_app_instance=6634zxcgfd, includepermalink=false}!!
Comments
SECTION: {section=local, subsection=education}!!!
INITIAL commentConfig: {includereply=true, canvas_permalink_id=washpost.com/8bvh5zpd9k, allow_comments=true, commentmaxlength=2000, includeshare=true, display_comments=true, canvas_permalink_app_instance=m6yzjj840m, display_more=true, moderationrequired=false, includefeaturenotification=true, canvas_allcomments_id=washpost.com/km4ey0dajm, comments_period=14, defaultsort=reverseChronological, includevoteofftopic=false, allow_videos=false, includesorts=true, markerdisplay=post_commenter:Post Commenter|staff:Post Writer|top_commenter:Post Forum|top_local:Washingtologist|top_sports:SuperFan|fact_checker:Fact Checker|post_recommended:Post Recommended|world_watcher:World Watcher|cultuer_connoisseur:Culture Connoisseur|weather_watcher:Capital Weather Watcher|post_contributor:Post Contributor, childrenitemsperpage=3, includeheader=true, includeverifiedcommenters=true, defaulttab=all, includerecommend=true, includereport=true, maxitemstop=3, source=washpost.com, allow_photos=false, maxitems=15, display_ugc_photos=false, includepause=true, canvas_allcomments_app_instance=6634zxcgfd, includepermalink=false}!!!

UGC FROM ARTICLE: {allow_comments=true, allow_photos=false, allow_videos=false, comments_period=14, comments_source=washpost.com, default_sort=, default_tab=, display_comments=true, is_ugc_gallery=false, max_items_to_display=15, max_items_to_display_top=3, moderation_required=false, stream_id=}!!!

FINAL commentConfig: {includereply=true, canvas_permalink_id=washpost.com/8bvh5zpd9k, allow_comments=true, commentmaxlength=2000, includeshare=true, display_comments=true, canvas_permalink_app_instance=m6yzjj840m, display_more=true, moderationrequired=false, includefeaturenotification=true, canvas_allcomments_id=washpost.com/km4ey0dajm, comments_period=14, defaultsort=reverseChronological, includevoteofftopic=false, allow_videos=false, includesorts=true, markerdisplay=post_commenter:Post Commenter|staff:Post Writer|top_commenter:Post Forum|top_local:Washingtologist|top_sports:SuperFan|fact_checker:Fact Checker|post_recommended:Post Recommended|world_watcher:World Watcher|cultuer_connoisseur:Culture Connoisseur|weather_watcher:Capital Weather Watcher|post_contributor:Post Contributor, childrenitemsperpage=3, includeheader=true, includeverifiedcommenters=true, defaulttab=all, includerecommend=true, includereport=true, maxitemstop=3, source=washpost.com, allow_photos=false, maxitems=15, display_ugc_photos=false, includepause=true, canvas_allcomments_app_instance=6634zxcgfd, includepermalink=false}!!
Show Comments