Diane Rennay Williams, an itelligent, attractive, unmarried 24-year-old employe of the Department of Justice, did just what the experts advise when her married boss made what she charged was an unwanted sexual advance: She told him to stop, and when he persisted, she filed a complaint against him.

Up to that point, her actions were just what other witnesses yesterday told a House Civil Service subcommitee on investigations that subordinate female empolyes should do when they are confronted by sexual harassment. But what followed in Williams' case isn't likely to encourage other women to step forward and blow the whistles on their bosses.

Within nine days of filing her complaint with an equal employment opportunities officer at Justice, Williams was fired. And when she sought to have her complaint investigated, officials at Justice argued that she had no grounds because she was no longer a government employe.

That was in September 1972. But Williams persisted, and three years later, a federal judge ruled she had been fired illegally. The decision was hailed as a landmark, the first time the sex discrimination provisions of the 1964 Civil Rights Act had been invoked successfully by an employe claimng sexual harassment. The judge ordered Justice to award her $19,147 in back pay.

"But recent developments make me feel like the defendant," Williams said yesterday to Committee chairman James M. Hanley (D-N.Y.). The 1976 lower court decision was overturned last year by an appeals court, which ordered a trial that has not begun. So seven years after she was fired, Williams is still trying to restore her good name in the federal record.

Hanley said the hearings are focusing on "a serious abuse of power, sexual intimidation by a male supervisor of a subordinate female employe." He said studies indicate that 40 to 70 percent of working women will encounter some form of sexual intimidation during their careers.

Hanley said that "a boys-will-be-boys atmosphere will be not condoned" in the federal government. He said the hearings, which are to be resumed Nov. 1, will attempt to define sexual harassment, initiate a survey to see how pervasive the problem is, and establish a reliable system for settling grievances.

Donna Lenhoff, a lawyer with the Women's Legal Defense Fund, cited a definition proposed by the National Orgnization for Women and the Working Women's Institute: "Sexual harassment is any repeated or unwarranted verbal or physical sexual advance, sexually explicit derogatory statements, or sexually discriminatory remarks made by someone in the workplace which is offensive or objectionable to the recipient, or which causes the recipient discomfort or humiliation or which interferes with the recipient's job performance."

Helen Lewis, executive director of the D.C. Commission for Women, said a policewoman offered this guideline: "A coworker can ask all he wants if he's not in a positon to threaten my job, unless it comes to a matter of threatened rape. But the first time my supervisor asks, that's sexual harassment."

Williams said that men in her office at the Community Relations Service at Justice "considered it a game, all in fun," and that some of her female co-workers "didn't sympathize with my plight. They said, 'Face it, who would complain about getting a promotion a year early?'"

At the time the judge ruled in Williams' favor, her boss, Harvey Brinson, a 40-year-old, married father of four, complained that the decision "gives the impression that I'm not denying it. . . I never did it (harass Williams)."

Yesterday, Brinson, who remains in his GS 15 postion as a public information officer at Justice, said in a telephone interview, "If anyone made sexual advances, it was Diane. We had an affair, during which she tried to get me to give her a promotion. I broke it off because I couldn't afford the motel bills."

Williams told the subcommittee that Brinson has "amended his earlier protestation of complete innocense."

She suggested that he has changed his statements because other women have volunteered to testify about him at her trial.

"He's trying to make me look like the disco queen of the city," said Williams, who is now 31, a second-year student at George Washington University Law School and part-time federal employe. "I have been forced to get my mother to attest to my social activities, to be my alibi."