Wiley A. Branton, a lawyer who gained prominence in major civil rights cases, has resigned after five years as dean of Howard University Law School, Howard President James E. Cheek announced yesterday.

Branton, 59, said he was leaving the school in "good shape" and was "anxious to return to private practice." He was praised by Cheek for "a distinguished contribution to legal education and to the university."

In the decade before Branton took office, Howard's law school was troubled by student unrest, bitter faculty disputes, and threats to its accreditation. Yesterday faculty members credited Branton with restoring stability and raising standards for admissions and graduation, although Branton said he is concerned that the pass rate of Howard law graduates on the bar exams remains relatively low.

During the past few months accreditation of the law school was reaffirmed by the American Bar Association and the Association of American Law Schools.

"I think we placed more emphasis on academic achievement under Branton and imposed more rigorous standards," said Prof. Herbert O. Reid, who has served on the faculty for 36 years. "We also involved ourselves more in civil rights matters than we had for some time."

A native of Pine Bluff, Ark., Branton was chief counsel for the black students who integrated Central High School in Little Rock in 1957. Later, he represented Freedom Riders in Mississippi and blacks involved in voter registration drives throughout the South. He came to Washington in 1965 as executive director of the President's Council on Equal Opportunity under President Lyndon Johnson. Afterward, he was executive director of the United Planning Organization, Washington's antipoverty agency, before entering private law practice here.

The Howard law school had seven deans in the 17 years before Branton took office. Its reputation as a training ground for top black lawyers declined as Southern schools, previously closed to blacks, were desegregated, and other law schools began to recruit them under affirmative-action programs.

In the early 1970s, the proportion of Howard law graduates passing state bar examinations across the country averaged 61 percent, compared to a pass rate of about 75 percent for all accredited law schools nationwide.

He said the law school still is having difficulty attracting top-flight applicants, but in the last two years, he said "we have made it much tougher to get in" even though this has caused a drop of about 15 percent in the size of the first-year class.

Also, Branton said the law school is strictly enforcing its requirement that all students must pass a proficiency exam in writing skills and reading comprehension. Last year, he said, about one-third of the entering students did not and were required to take a noncredit course in communications skills. They must repeat the exam, he said, until they pass it.

Branton said the faculty has adopted a stricter grading policy, which will go into effect next fall, and has agreed to go ahead with comprehensive examinations. These will be given on a trial basis for the next two years, Branton said, and then will be required at the end of the second year in 1986.

"Some of these things flowed from the critical accreditation reports," Branton said, "and some from our own evaluation of what must be done . . . We're committed to turning out lawyers who can pass the bar exams and serve the public well."

Branton disclosed yesterday that last winter he had criticized the decision of Howard administrators to expel the undergraduate editor of the student newspaper. He said he also disagreed with the contention of university attorneys in two discrimination cases brought by whites that as a predominantly black university Howard may "take race into account" and give preference to blacks in hiring and promotions.

"I just think that's wrong," Branton said yesterday. "You can't have your cake and eat it, too. I don't think a black school has any more right to discriminate in favor of black people than other schools have to discriminate in favor of whites . . .

"With the historic role of Howard law school in civil rights, you might think we would have been consulted by university attorneys ," Branton continued, "but we had nothing to do with that argument at all."

The racial preference argument was rejected in both cases. Last week Michael R. Winston, who became Howard's vice president for academic affairs in mid-May, called the argument "an aberration."

Alan Hermesch, a university spokesman, said Branton would leave his post Sept. 2.