The Justice Department charged yesterday that the state of Maryland violated the 1964 Civil Rights Act by reserving four high-level positions at the state lottery agency for women and minorities, thereby discriminating against white males.

The lawsuit, filed in federal court in Baltimore, is the third "reverse discrimination" suit brought by the Justice Department against state and local governments in a year and appears to represent an increased interest in such cases by department officials, some civil rights lawyers said.

The case stems from a five-year-old complaint filed by 10 white males who had applied for the state lottery jobs in 1983 and were rejected.

The Justice Department suit states that the applicants were rejected solely because of a discriminatory "employment plan" adopted at the direction of then-Maryland governor Harry Hughes (D).

The suit was immediately criticized by Hughes and state officials as a waste of time and resources and an inappropriate use of the civil rights laws. Acting at the behest of the Maryland Human Relations Commission, which found that there weren't enough blacks and women at upper levels of the lottery agency, the state created four new posts -- including a deputy director and an area manager -- and filled them with two black men, one black woman and one white woman, officials said.

"We were trying to do something that was the right thing and so we wouldn't get sued at the time," Hughes said.

"I suppose it's not unexpected from this Justice Department. I wonder whether they would have brought suit if the lottery agency was not open to employment and if a black or woman had brought it to their attention. In this Justice Department, I doubt it."

Attorney General J. Joseph Curran Jr. promised to vigorously defend the state against the Justice Department suit.

He said that Hughes' actions were right "in that climate of 1983 and in light of the fact that the Human Relations Commission had cited the Lottery Commission."

Department officials said the suit illustrates their determination to enforce the civil rights laws as they were originally intended, without regard to race, creed or sex.

"We don't throw complaints in the trash can because it's white males who were the victims," said Roger Clegg, deputy assistant attorney general for civil rights.

In this case, they said, the aggrieved white males filed their complaint with the Equal Employment Opportunity Commission, which in September 1988 found "reasonable cause" to believe the charges and referred them to the Justice Department.

"We handled this the same way we handled all cases referred to us by the EEOC . . . . We investigated it and determined that a lawsuit had to be brought," Clegg said.

The suit was brought under Title VII of the 1964 Civil Rights Act, a section that traditionally has been used to combat discrimination against women and minorities.

Throughout the Reagan administration, the Justice Department occasionally used the law to bring suits on behalf of white males, a practice that has continued under Attorney General Dick Thornburgh.

During the past year, for example, the department has brought suits against the city of Allentown, Pa., alleging that it discriminated against white men in hiring for its police department, and West Haven, Conn., alleging that it discriminated against men in hiring for clerical jobs and against women in hiring for skilled craft positions.

Both suits were settled with consent decrees in which the municipalities agreed to prevent any discriminatory conduct in the future.

The suit against Maryland asks the court to prevent Maryland from engaging in discriminatory conduct as well as to pay "remedial relief sufficient to make whole" all the white complainants who were turned down for the jobs.

State officials said that could prove difficult because only two of the complainants are still "viable" candidates for the lottery jobs. Dennis M. Sweeney, deputy Maryland attorney general, said that in a recent effort to persuade the Justice Department not to bring the case, state officials had offered to pay a total of up to $11,000 to the two men who were prevented from being considered for the four slots.

That offer, he said, was rejected by the Justice Department.

"We're going to spend that much to do this archaeological dig that it will take in this case," Sweeney said.

"But they want us to sign a consent decree saying we did wrong and won't do it again.

"We said, 'Do you have evidence that the state has done anything like this since 1983?' And they said, 'We don't know.'

" And we said, 'Do you have any complaints?' And they said, 'That's not our responsibility.' "