Federal Judge Gerhard Gesell will hear accusations in court here today that the Air Force is waffling on its policy towards homosexuals in a landmark gay rights case involving an airman who was kicked out of the service five years ago after he openly declared his homosexuality.

Lawyers for Leonard P. Matlovich, 37, a decorated Vienam veteran with a spotless service record, have charged that the government had "trifled" with a U.S. court order requiring it to define conditions under which gays may serve in the Air Force.

In the past year, the Air Force has filed several contradictory statements, some saying there are posts in which gays may serve, others saying there are none.

The gay rights case involving Matlovich began in 1975 when he disclosed his homosexuality in a letter to the secretary of the Air Force. Under Air Force regulations, which prohibit homosexuality unless "the most unusual circumstances exist," the Air Force discharged Matlovich.

Claiming his discharge was unconstitutional and that the government had not explained "unusual circumstances," Matlovich sued the Air Force.

In July 1976, Gesell upheld the Air Force's ouster of Matlovich, but urged, the service to reexamine its anti-homosexuality policies that led to his dismissal.

"This is a distressing case," Gesell said at the time. He said the Air Force "might well be advised . . . to move to a more enlightened policy in light of the more open atmosphere" about the subject of homosexuality. However, Gesell said he ruled in favor of the Air Force because there is no constitutional right to engage in homosexual activities.

Matlovich appealed his case to the U.S. Court of Appeals, which ruled in December 1978 that the Air Force had not properly distinguished the exceptions under which a gay serviceman could be retained, and why Matlovich did not qualify. The case was sent back to Gesell and the Air Force was requested to come up with an explanation of its policy.

In September 1979, in response to the court's order, Secretary of the Air Force Hans M. Mark stated that the Air Force does not recognize exceptions to the policy of discharging confirmed, practicing homosexuals."

However, last week Mark filed court papers saying his previous statements were "incorrect" and that Air Force Regulations do contain "an exception provison" which was potentially available to Matlovich. However, the Air Force found no basis to grant Matlovich an exception.

Government lawyers would not comment on the Matlovich case. However, the Air Force said that a change in policies toward gay servicemen is not likely. aIt is not possible, a government brief submitted yesterday argues, "to predict the eventual impact of changes in the civilian sector upon the military" and the exception provison provides sufficient flexibility for the Air Force's policy toward gays.

Matlovich's attorneys said in court papers that the Air Force has conceded that it is "unable to explain the actual reasoning" behind Matlovich's discharge. As a result, Matlovich should be reinstated they argued.