President Carter yesterday nominated Virginia's highest-ranking black judge for a seat on the federal bench, setting the stage for a long-brewing showdown with Sen. Harry F. Byrd Jr. (Ind.-Va.)

"I can't imagine anything worse for the American people than to have a quota system for federal judges," Byrd declared immediately after the White House announced the nomination of Richmond Circuit Court Judge James E. Sheffield to become the first black federal judge in Virginia since Reconstruction.

Byrd charged Carter had decided "to force an unjustified confrontation" over the selection of judges and accused the president of using the judiciary "to advance his own political purposes."

Sheffield's selection also appeared likely to pose the first test of the Senate Judiciary Committee's recently revised method of confirming judicial nominees.

Traditionally, the Judiciary Committee allowed senators from any state to block nominations of federal judges in their state. Under this previous system, Byrd would have been permitted to prevent Sheffield's confirmation merely by withholding his approval.

Sen. Edward M. Kennedy (D-Mass.), the Judiciary Committee chairman, has announced, however, that this longstanding practice will no longer be honored.

Sheffield, 47, declined yesterday to comment on the controversy surrounding his nomination saying any statement would be "highly inappropriate."

"It's a difficult position that I'm in," Sheffield remarked in a telephone interview. "The least I say now, the better." Sheffield was chosen as Virginia's only black state Circuit Court judge in 1974.

The White House yesterday also nominated J. Harry Michael Jr., 61, a Charlottesville lawyer and Democratic state senator, for a federal judgeship in Virginia. Both Sheffield's and Michael's nominations had been expected since last year, but Sheffield's selection was far more controversial because of Byrd's opposition.

The judgeship battle between the Carter administration and Byrd has been simmering since December 1978, when Carter threatened to leave four newly established federal judicial seats vacant unless Byrd agreed to recommend blacks and women for judgeships. Byrd immediately refused.

Byrd had submitted two lists of 10 judicial nominees to the White House. All were white males. Byrd argued then that he had done "precisely what the president asked me to do" by relying on special judicial selection panels to pick candidates for judgeships -- a point he reiterated yesterday.

After the initial clash, however, Carter appeared to bow to Byrd by moving to nominate three white male judges recommended by Byrd. The White House first nominated two Byrd-supported candidates for judgeships, Richard L. Williams of Richmond and James P. Jones of Abingdon. In addition, Michael, who was also on Byrd's lists, had been considered a prospective nominee since August.

The dispute between Byrd and the White House led to a Judiciary Committee stalemate. The committee delayed action on the two judgeship nominations supported by Byrd, indicating it wanted to consider all four Virginia nominees at one time.

It was unclear yesterday how the Judiciary Committee would respond to Byrd's opposition to Sheffield. A key question appeared to be whether Byrd would vigorously challenge Sheffield's candidacy, or instead, would decide to put up only a pro forma fight. A Byrd aide said he did not know what steps Byrd would take.

A White House official said Carter "hopes and expects" that the Judiciary Committee will confirm Sheffield's nomination, but declined to elaborate.

In his statement yesterday, Byrd asserted that the president's "credibility has been greatly damaged" by his decision to nominate Sheffield.

"In a hand-written letter to me two years ago," Byrd said Carter ". . . urged that federal judges be appointed on the basis of merit. He stated that the way to remove the selection process from politics was by utilizing broad-based commissions to screen and to recommend.

"I did precisely what he asked, and in the way he suggested it," Byrd added, noting that blacks and women were included as members of his judicial selection commissions.

"A confrontation with the president of the United States is never pleasant. But I shall stand firmly behind the [judicial selection] commissions' recommendations," Byrd said. "I doubt that either the president or Virginia politicians seeking to make a political football of judicial appointments are likely to gain in the eyes of the people of Virginia."

Byrd's reference to Virginia politicians appeared to be a swipe at Rep. Herbert E. Harris II (D-Va.), who has campaigned for more appointments of blacks and women to the federal bench. Shortly before Sheffield's nomination was announced by the White House yesterday, Harris issued a statement hailing it as "an important and historic benchmark to the judicial progress of Virginia."

Byrd's assertion that he complied with Carter's plan for choosing federal judges also prompted criticism yesterday.

"There really is an abuse of process there," charged Ann Macrory, former executive director and currently a board member of the Judicial Selection Project, a private Washington-based organization that advocates broader representation of blacks, women and public-interest lawyers on the federal bench.

Macrory argued that although Byrd's judicial selection commissions included blacks and women, they failed to carry out "affirmative outreach efforts" to search for black and female candidates for judgeships. "The commissions just went to the traditional sources," she said.

Similarly, Elaine R. Jones, assistant counsel for the NAACP Legal Defense Fund Inc., contended, "Sen. Byrd says he did [comply with Carter's plan] when, in fact, there was no outreach to blacks and women." She praised Sheffield's nomination.

Sen. John W. Warner (R-Va.) said he was "reserving judgment" on the judgeship nominations because, as a Republican, he had had "no involvement" in the selection of judicial nominees.

Ironically, Byrd also might have been excluded from any role in helping to choose judicial candidates -- but for what the White House conceded was an error. Byrd votes as an independent, rather than a Democrat. But the Carter administration mistakenly soliciated his advice on prospective judges for Virginia.

According to the Justice Department, there are 40 black federal judges in the United States. U.S. District Court judgeships -- those at issue in Virginia -- now pay $54,500 a year.

The Judiciary Committee recently showed willingness to depart from tradition by rejecting Carter's nomination of Charles B. Winberry Jr. for a federal judgeship in North Carolina. It was reportedly the first time in 42 years that the panel had taken this step.

In contrast with the Sheffield controversy, however, Winberry was backed by a Democratic North Carolina senator and was rejected after a controversy over allegations that he had breached professional ethics.