A federal judge in Richmond yesterday rejected the Justice Department's proposed settlement of a lawsuit over alleged discrimination in the Virginia highway department, holding in a stinging decree that the state and federal governments were proposing to discriminate against white males.

U.S. District Judge D. Dortch Warriner, a Southside Virginian who has tangled with Justice Department lawyers in previous lawsuits charging discrimination, ruled that the department was trying to impose hiring quotas on the state in the guise of goals. He said it was "saddening" that Virginia, which has a long history of contesting federal discrimination suits, had concurred in the proposed settlement.

"Calling a quota a goal is like calling the tail on Mr. Lincoln's sheep a fifth leg," Warriner said in rejecting the settlement. "It is still a sheep's tail . . . . As far as white males are concerned, the court has nothing in the record to indicate whether this class is willing to suffer the discriminatory impact of the consent decree."

Warriner also said he could not accept the decree because the Justice Department had furnished no evidence to support its claims of discrimination and because he was unfamiliar with the reputations of the government lawyers on the case.

"Experience in other cases has shown that the mere fact that one is a government lawyer is no guarantee that he is an able, experienced, knowledgeable, and dependable lawyer," Warriner said.

Justice Department spokesman John V. Wilson declined to say what the government would do next, although he noted that Warriner's ruling could be appealed. Wilson was quick to respond, however, to the judge's questioning the credentials of Katherine F. Ransel, the department's chief attorney on the case, and her two associates, Teresa D. Johnson and Marybeth Martin.

"They're very experienced," Wilson said. "Katherine is one of the senior lawyers in the area of employe discrimination . . . . She handled the Fairfax County case, which was the biggest back-pay award the department has ever won in a public employer-employe discrimination case."

Virginia officials also declined to predict the next step in the case. State Highway Commissioner Harold C. King said that the proposed settlement was "preferable to the possibility of years of disruptive litigation and the costs such litigation could involve."

King also indirectly rejected the judge's charge that recruitment goals would discriminate against white males.

"The proposed settlement represented a continuation of this department's efforts to follow equal employment principles for all, and it was negotiated in the best interest of all the citizens of the Commonwealth of Virginia ," King said.

The Department of Justice sued the highway department on Dec. 30 after a five-year investigation, alleging that the state government's largest agency long had discriminated against blacks and women in hiring and promotions. While denying any discrimination, Virginia agreed to settle the lawsuit the day it was filed, as Democratic Gov. Charles S. Robb attempted to steer the state away from its tradition of confrontation on racial issues.

The settlement required the state to pay up to $1 million to blacks and women found to have been discriminated against and to set hiring goals in each of the department's nine districts, including Northern Virginia. The department also promised to create training programs for minorities and to replace what the Justice Department called the "subjective and discretionary employment standards" that Virginia had used.

The highway department lawsuit is the second that the Robb administration had tried to settle during its first year. A similar Justice Department suit against the state police department, which had been contested by two previous Republican governors, was settled earlier last year.

Warriner also was the judge in that case in 1977 and 1978 and ruled against the Department of Justice, forcing the government to appeal twice. Virginia employed only 20 blacks on its 1,100-officer force at the time that the case went to trial and acknowledged in court that the names of black applicants, while not listed by race, were marked with a red comma on recruitment lists. Warriner ruled, however, that the federal government had failed to prove "a pattern and practice of discrimination."

The Justice Department said in its complaint that 10.3 percent of the highway department's current work force is black and 11.5 percent is female, and that black workers are concentrated in menial jobs. The consent decree that Warriner rejected said that the district-by-district recruitment goals "are not and will not be treated as quotas." Failure to meet a goal, the government said, would not in itself constitute a violation of the agreement.

The promise did not mollify Warriner, who said that a preference for quotas in hiring is "deplorable and was, at one time at least, unconstitutional." He said that the proposed settlement violated the Fourteenth Amendment to the Constitution, which guarantees equal protection for all citizens.