The University of Virginia broke from tradition yesterday by going outside the state and picking as its new president Robert M. O'Neil, president of the University of Wisconsin and a legal scholar with no ties to Virginia.

The selection of O'Neil, a Harvard-educated lawyer described as a political liberal, was part of an effort by the university to enhance its prominence nationally.

O'Neil, 50, will be the university's sixth president when he takes office Sept. 1. He succeeds Frank L. Hereford Jr., who announced last year that he wanted to resign the presidency of the Charlottesville institution after 11 years to return to teaching physics.

The election of O'Neil by the University Board of Visitors reflects a dramatic shift in thinking at an institution that for the past half-century has been parochial in its choices for the presidency. Only O'Neil and the university's first president, named in 1904, have been chosen from outside Virginia for the prestigious position.

"The day for a national leader had arrived," said Kenneth Thompson, a member of the faculty search committee that was closely involved in the selection process.

"Everyone said . . . we ought to have a national educational leader who would symbolize what has happened to the university."

University officials would not disclose O'Neil's salary. Hereford was paid $76,000 annually.

O'Neil, educated at Harvard College and Law School and a clerk for Supreme Court Justice William J. Brennan Jr., is described by a former colleague as a "solid liberal" whose political views will be somewhat to the left of the Virginia political mainstream.

"It is an extraordinary choice by a board as conservative as this board is," said one university source close to the selection process. As president, O'Neil will have to deal with the Virginia legislature, considered by some to be an old-boy network bound by conservative political traditions.

Fred G. Pollard, university rector and head of the board's selection committee, said O'Neil was selected in part for his commitment to the liberal arts. He called O'Neil "a distinguished scholar and writer by any measure."

O'Neil said yesterday that he lamented the emphasis on professional subjects in undergraduate education and what he called its "detrimental" effect on liberal arts studies.

The incoming president, who also will teach in the law school, said he would propose dual minors and majors in liberal arts for undergraduates majoring in fields such as business, education, engineering and architecture.

O'Neil's academic specialty, First Amendment law and his extensive writings on public issues such as affirmative action, made him particularly appealing to the search committee, members said.

The university has been plagued in recent years by racial tensions and criticized for its failure to attract minority students.

O'Neil was described yesterday by one faculty member as a "political pragmatist" with a knack for resolving disputes.

Hereford, while lauded for raising $137 million for the university during its most successful fund-raising drive, caused an uproar on campus in 1975 when he refused to resign from a segregated country club.

He was censured by a university faculty group before quitting the club.

O'Neil was selected after a 10-month search in which 312 candidates were considered. He was one of a handful recommended by both the student council and the faculty senate.

He began his career in 1963 on the law faculty at the University of California at Berkeley.

In 1972 he entered academic administration at the University of Cincinnati, and later served at Indiana University before going to Wisconsin in 1980.

There he managed a statewide system of 13 universities and 13 two-year junior colleges with a $1.3 million annual budget.

O'Neil is married and has four children.