Free and open expression, the hallmark of academia, is being tested in federal court by a topic that, since medieval times, has been largely taboo for public discussion: sex and human sexuality.

At issue is the often-secretive, closely guarded ritual of granting university tenure, a process that frequently collides with the lofty goal of preserving academic freedom and a professor's right to say what he thinks.

Also involved is the little-known but growing field of sexology, which is threatened nationally by a rising tide of conservatism as it strives to study human sexuality in the most neutral way.

The case started when Prof. Roger Libby, 43, a well-known and widely published sexologist, was denied tenure by the University of Massachusetts. On Feb. 11, he filed a $165,000 suit against the school in U.S. District Court in Springfield, Mass.

Libby teaches that there are a multiplicity of sexual life styles, that monogamy is not necessarily good for everyone and that masturbation, perversion and premarital sex are legitimate topics of inquiry.

The case, for which a hearing date has not been set, could prove significant for the fledgling field of sexology and could provide a rare open forum on the mysteries of how and why tenure is granted or denied.

In interviews, some noted sexologists said the case is significant because the conservative group Accuracy in Academia has announced plans to monitor college classrooms for disinformation and bias.

Sexology is particularly vulnerable to attacks by such New Right groups, the sexologists said, as some within the discipline seek to delve into such sensitive areas as homosexuality, child molestation and masturbation.

"This is probably just the tip of the iceberg," said Dr. John Money, an endocrinologist and expert in early childhood sexual development at Johns Hopkins University Medical School in Baltimore. "There's a very powerful antisexual movement going on at the present time."

Dr. Lynn Atwater, a sexologist and associate professor of sociology at Seton Hall University, said, "Sex is just a highly charged issue in our society."

Libby, in an interview, said, "I'm pursuing this case because it's symbolic of what's happening to people in the sexual field at universities across the country."

Libby said he was hired six years ago as an assistant professor in the University of Massachusetts' home economics department and was told to develop a graduate-level center dealing with family issues. From the start, he said, he was not permitted to establish the program as he saw fit. He said that he had no graduate assistants or travel budget and that officials ruled against making peranent one of his proposed courses, "Sexuality and Sex Roles."

University officials have declined to comment on the suit because the case is in litigation. A university spokeswoman, Jeanne Hopkins Stover, also said that tenure disputes are considered personnel matters and are strictly confidential.

Libby, who received his doctorate in sociology from Washington State University, is a founding editor of the academic journal Alternative Lifestyles, which recently changed its name to Lifestyles.

He wrote the forward to the international best seller "Open Marriage" and a human-sexuality textbook called "Sexual Choices." A fellow sexologist, Dr. Ira Reiss of the University of Minnesota, said Libby "is a man who's respected as a researcher and a writer in the area."

Libby approaches his topic from a liberal perspective, and his views and sometimes raw sense of humor are often controversial. "I advocate a liberal, open approach to sex," he said. "The whole idea of looking at a contemporary family is not just to look at the nuclear family but at people living in communes, single people and other types of nontraditional families."

He said he never sought to advocate a particular life style to his students but only to alert students to the existence of alternatives.

"Roger takes what, by conservative university standards, would be considered a radical or provocative stance," Lifestyles editor Larry Constantine said.

Libby said his courses, and his sexually oriented jokes in class, have been popular with students. "The students gave me the best evaluations," he said.

In the court suit and in an interview, he contended that he ran afoul of faculty colleagues in the university's traditionally conservative home economics division.

Sexologists interviewed at other campuses said courses dealing with sex typically are popular with students but derided by other faculty members. Part of the problem, they said, is that few schools have separate divisions for sexology and usually offer it as part of home economics or sociology programs, whose professors often resist teaching it.

"A course like that is a moneymaker," Reiss said. "A freshman class in sexuality can attract hundreds of kids. But university officials would like to have someone with their own values teaching it. Presenting alternatives and not condemning the less popular ones gets people upset."

According to the typical university tenure track, a professor is hired for six years and considered for tenure in the sixth year. When Libby's case came up, a tenure panel in his division denied him permaent status, but a college-wide tenure panel reversed that decision.

Libby contends that the college-wide panel then reversed itself and voted to deny him tenure under pressure from ranking university officials.

Libby's suit specifically names the university trustees, President David C. Knapp, Chancellor John Duffey and Provost Samuel Conti. Stover, the university spokeswoman, said none would comment on the case because it involves a pending court suit and because they consider it a private matter.

Other sexologists said that Libby's case is not unusual and that sex researchers nationwide are often scorned and harassed, their motives suspect, their discipline considered "weird." In a politically conservative climate, they said, research funding is difficult to find for such sensitive topics as sexual perversions or extramarital sex.

"As the tenor of the times has changed, there is less leeway in the classrooms to talk about what is really happening," Atwater said. She teaches a course called "Sociology of Sexuality" but she said she began teaching it after receiving tenure.

"A stigma is applied to those of us who teach or research sex," Atwater said. "Either we're perverted, we're sexually repressed, or we're oversexed and this is just another outlet for us. Everybody who works in sex research has felt this stigma."

Because of taboos dealing with sex, much-needed sex research goes wanting, according to those interviewed. Money said research into the early sexual development of children, which he said could help prevent future sexual disorders, is never conducted.

Meanwhile, such topics as child abuse, generally deemed antisex, receive ample government funding. Money cited a noncompetitive Justice Department grant to a researcher to study the depiction of children in such magazines as Playboy and Penthouse and its possible link to child-abuse cases.

Money has written "The Destroying Angel," a history of sexual taboos. In it, he traces antisex sentiments from the Spanish Inquisition, which he describes as an antisex campaign as much as a purge of suspected religious heretics, through Nazi Germany and the specter of troops destroying Germany's Institute For Sex Biology and persecuting sex researchers as "pornographers, homosexuals and Jews."

"I think that's sort of a lesson of history," Money said. "Because of the great pervasiveness of the sexual taboo, people in political and religious authority get a great deal of power by controlling it."

He added, "The loudest voices being raised at the present time are those of the New Right, and the antagonism to sex is related to antiabortion, anticontraception, antisex education and antihomosexuality."