NINE YEARS AGO, an English instructor at the University of Hawaii named Joan Abramson decided she was the victim of sex discrimination. She had been denied tenure and men with less advanced degrees and fewer published works were being promoted over her. She filed a complaint. As new laws against discrimination went into effect, she filed more complaints.

She lost her job. Since then government agencies have repeatedly found she was discriminated against and that the university fired her in retaliation for complaining. But so far, all Joan Abramson has gotton out of it is a lot of aggravation and a box full of files.

Abramson's story is complicated. She has pursued her case through several different federal agencies, including the Health, Education and Welfare Department, the Justice Department and the Labor Department. Each of these agencies has ended up agreeing that she was discriminated against, yet for a variety of reasons none of the agencies has been able to prosecute her case vigorously

Her story is a classic illustration of how the federal mechanism for stopping sex discrimination did not work when confronted with an obstinate institution that refused to conciliate and change its ways.

Abramson is 47 years old, the wife of a professor at the University of Hawaii, and the mother of three children, two of whom grew up with what was known in the family as "the case." She went to work at the university in the late '60's after her husband joined the faculty, and everything was fine until she was denied tenure.

Both the university personnel committee and the faculty senate committee decided she should have tenure and the university president's commission on the status of women found that sex had been a factor in denying her tenure. Nevertheless, the university president went along with the English department's recommendation and Abramson was told the issue was closed.

Not quite, Abramson filed an equal pay complaint with the Department of Labor in September of 1972 -- and was fired within two weeks. The Department of Labor obtained her reinstatement with a 53 percent pay increase, but within a month the university sent her a notice of termination that would go into effect next year.

Next, there was a dispute about whether or not she had a contract. Finally, the university succeeded in firing her in September of 1973 by abolishing the program she was connected with.

By then Abramson had taken her complaint to the Equal Employment Opportunity Commission, which decided in 1974 that she had been both discriminated against in her employment and retailated against after her complaint. The EEOC tried to reach an amicable settlement with the university and failed.

The EEOC then referred her case to the Justice Department, which also found that Abramson had been discriminated against and gave her a "right to sue" letter. The Justice Department had the authority to file suit against the university, but Abramson says she was told the department at that time did not have the money for such individual cases.

Meanwhile, a complaint that she had filed in 1971 with HEW office of civil rights was getting nowhere. In 1975, Abramson says, she testified before a congressional subcommittee on the difficulties she was having getting HEW to act on her case. The subcommitte staff report focused on her testimony, says Abramson, and within a year HEW sent her a letter informing her that they had found no discrimination.

She appealed this finding to the office of federal contract compliance programs, in the Labor Department, which three years later found, again, that Abramson had been discriminated against.

The Labor Department invited the university to settle the dispute informally.Those efforts broke down in December when the university informed the Labor Department that it would not consider reinstating Abramson in any capacity.

The Labor Department then drew up a conciliation agreement in which the university was asked to rehire Abramson as a full professor, grant her tenure retroactive to 1971, reimburse her for all money lost as a result of her treatment, and pay her legal fees.

The agreement also detailed various professional considerations she was to be accorded and forbade the university from further retaliation against her. The agreement was accompanied by a tough letter telling the university to come to terms or go to court. So far, neither has happened.

The case subsequently was referred to the solicitor's office, where a court case was to be prepared.

"The last straw for me," says Abramson, "came when I found out that the solicitor assigned to prepare the case said it should be reopened for investigation. Half of the people are retired. Some are dead. Everything is in sworn depositions. There is not a rock that hasn't been turned over in that case. It's not ongoing. It ended in 1973 when I was terminated."

Abramson was in Washington last week and talked with James Cisco, head of the Department of Labor's program operations division for the contract compliannce office. He says his office will make one more attempt at conciliation, and if that fails the case will go to an administrative hearing. The university, should it lose the case in the Labor Department, then can take it into federal court. Durward Long, the university chancellor, declined to discuss the university's treatment of the Abramson case on the grounds it is still in litigation.

Abramson contends that the university capitalized on HEW's lack of commitment toward academic discrimination cases in the early '70's and subsequently has used such delaying tactics as shifting attorneys to keep the case out of court.

She has supported herself over the years by writing four books, including two on discrimination cases. She says you have to have a sense of humor to persevere in a sex discrimination case and you also have to have moral and financial support.

"Unfortunately," she says, "you can't engage in one of these cases unless you are a kept woman or married."

Abramson's experience has profoundly changed her mind about what women should do when they feel they are victims of discrimination. When it happened to her, she fought back and tried to get the federal government to back up its commitment to equality by helping her.

Nine years later, with virtually nothing to show for her efforts other than a lot of official papers telling her that her cause was just, she is giving women another message as she travels around the country speaking or promoting her books.

"I tell them, if you can, just get another job," she says.

That's not the way it was supposed to be.