A hearing officer has recommended that the Navy court-martial the captain of a Florida-based frigate accused of having a homosexual affair with one of his crewmen during a Mediterranean cruise last spring.

The officer recommended the trial for Cmdr. Gerald M. Vanderwier, 42, a decorated veteran of the Vietnam War, after a two-day hearing last month in Mayport, Fla. Vanderwier, who was relieved as skipper of the USS Edward McDonnell in June, is charged with sodomy and conduct unbecoming an officer, specifically, fraternization with an enlisted man.

Vanderwier, through his attorneys, has denied the allegations of homosexuality.

A Navy spokesman said yesterday that he knew of no other recent case in which the captain of a Navy ship has lost his command over allegations of homosexuality. "To my knowledge, it's a first," said Lt. Tom Yeager, who added he has not made a complete check of naval records.

Vice Adm. Edward S. Briggs, head of the Navy's Atlantic surface forces, is expected to decide soon on how to handle the case. Briggs' options range from court-martial to administrative action to a dismissal of the charges, according to a spokesman for his command in Norfolk.

"If he decides for a court-martial, he will also decide where it will be," Lt. Cmdr. Bob Prucha said.

Both the Navy and Vanderwier's attorneys said yesterday that hearings in Mayport July 19 and 20--called an Article 32 hearing under the Uniform Code of Military Justice and considered the equivalent of a probable cause hearing in civilian courts--centered on the testimony of Vanderwier's accuser, Petty Officer John Rainville.

Rainville, 21, a hospital corpsman, testified he had sex with Vanderwier on three different occasions--in a hotel in Tunis and aboard the McDonnell while the fast frigate was docked at Rota and Malaga, Spain, according to Prucha.

Vanderwier's attorneys produced medical records at the hearing showing Rainville had a history of mental instability and of drug and alcohol abuse. Rainville also said his diary contained references to other homosexual relations that never occurred, according to Dan Smith, an associate of Vanderwier's civilian attorney Henry Coxe.

"The crux of the case is Rainville's credibility," Smith said.

Suspicions about Vanderwier's relations with Rainville initially were raised when another sailor came across Rainville's diary and letters that had homosexual overtones, Smith said.

Lt. Cmdr. David Klich, second in command on the McDonnell, reported those suspicions to officers on the aircraft carrier Nimitz, which also was in the Mediterranean at the time.

The McDonnell returned to Mayport on May 20 and on June 6, Vanderwier was relieved of his command and reassigned there to the staff of Destroyer Squadron 12.

Vanderwier, who is married, enlisted in the Navy in 1959 and was commissioned after officer candidate school in Newport, R.I., in 1964. According to Smith, his client received several combat decorations, including the Bronze Star, while serving in Vietnam. Vanderwier assumed command of the McDonnell, which carries a crew of 220, in January of last year.

If the case goes to court-martial, Vanderwier could face dismissal from the service, forfeiture of pay and benefits and up to 15 years imprisonment at hard labor, a spokesman said.