PUBLISHERS sent an unprecedented 60-plus books related to homosexuality out into the world in 1977. They are spending the first quarter of 1978 awaiting the results. Encouraged by the phenomenal success of Rita Mae Brown's Rubyfruit Jungle, which in four years sold 70,000 copies without benefit of mass advertising for Daughters Press, a small feminist house, Bantam has printed a first run of 300,000 copies of the book. Crown, following the same formula that worked so well for The Joy of Sex, has released 75,000 and 50,000 copies respectively of The Joy of Gay Sex and The Joy of Lesbian Sex, and reports that together the books sold 45,000 copies before the official release date in October.Dell will bring out John Rechy's The Sexual Outlaw this month and plans a first printing of 400,000. In the first seven months of its publication the book sold 25,000 hardcover copies, a number that its publishers, Grove Press, who enjoyed the "breakthrough" success f Rechy's novel, City of Night, in 1963 (it sold 65,000 hardcover copies), seem to find disappointing.

City of Night, with its depiction f subculture of drag queens and hustlers that might now only be described in terms of commonplace cliches - sordid, shocking, underground - was the book that started it all. But it had taken over a decade for establishment houses to begin to regard gay books as anything more than an isolated phenomenon. Even today, the City of Night, as portrayed in Rechy's latest nonfiction book, is off limits for many booksellers. Major bookstores and department stores demurred on The Sexual Outlaw, and the distributor could place the book in only two cities in its Southeast region. This is the fate of a book, says Rechy's editor, that is "not only gay but explicit." He acknowledged, however, that the book had also been attacked in the gay press.

Other publishers are more sanguine about the publishing picture for gay books, even though they readily admit the market can't be quantified, and there are few generalizations to be made about it. Words used to characterize it are most often "vilatile" and "amorphous," but perhaps "heterogeneous" best sums it up. The field includes everything from detective and science fiction novels to "how-to" sex manuals to serious political collections to literary trade books. All agree that the readership is vast, unusually, literate and engaged; that it includes "straights" as well as gays, has a pronounced feminist element, and probably numbers as much as half-a-million for a quality book with a homosexual theme.

Like most paperback editors, Ace Books' Michael Seidman - who when he was a Signet commissioned Coming Out, a "liberated" gay novel that it rumored to have sold over 65,000 copies in 1977 - says he is only looking for a good mass-market novel. "If it has a gay theme, fine." But he also hopes to reissue The Sand Fortress by John Coriolane, a novel which consists of the often sexually explicit reminiscences and ruminations of a priapic schoolteacher, and try to pitch sales to a strictly gay male audience.

The mass-market approach is very much evident throughout the trade, as publishers attempt to build a readership as faithful to their own authors as Patricia Nell Warren's (The Front Runner, and still going strong in paperback; The Fancy Dancer, 200,000 from August through October) and Gordon Merrick's (The Lord Won't Mind, One for the Gods, Forth into Light, An Idol for Others). These novels are certainly gay and explicit, related to, but a far cry from, the tepid hellinism of Mary Renault. But they are also "a good read" and feature qualifiedly happy endings, a facile reversal of the last-chapter suicide that was an obligatory fixture of gay novels in the '50s. The contemporary gay liberation novel turns out to be not that different from the conventional paperback romance. That is one of its charms and, one must conclude, one of its prime selling points. (At the end of An Idol for Others, Merrick's hero Walter Makin dies, tragically, but liberated from his image as Father of the Year.) They are also books that most heterosexual readers can pick up at Brentanos or at the airport with as few qualms as they might experience purchasing the latest Victoria Holt - for they are very much in that vein. The conflicting philsophies that bring gay readers pulp romances like The Front Runner as well as acerbic feminist books like Rubyfruit Jungle reflect equally conflicting perceptions of the gay experience on the parts of publishers and gay readers.

William Morrow has just contracted three more books from Patricia Warren. Her editor Jim Landis attributes the explosion in sales of gay books to the fact that "there is now a literature, whereas there wasn't before, when, gay books fell into the '42nd Street' category. There has been a whole process of coming out. It seems to be happening literarily as well as culturally." Whether great gay novels have lain around in trunks for decades is a moot point, however. One has only to compare the publishing scene of a decade ago, when gay books were the province of small mail-order houses and certain out-of-the-way bookstores, with the situation now to conclude that mass marketing, far more than struggle sin Dade County and elsewhere, accounts for the revolution in gay books.

Gay people are not only increasingly willing to declare an openly gay lifestyle, they are willing to spend money to enhance it. Marian Behrman, the publicist for Crown's "Joy" books - affectionately, it seems, referred to as Joy One, Two and Three - believes the theory that male and female homosexual readers were simply waiting for the same thing heterosexual readers were waiting for before The Joy of Sex (Joy One) was published: "Of course, people were doing all this long before Joy One. It sold because a respected medical authority said, 'It's O.K. I give you my permission.'"

It seems doubtful that the more sexually sophisticated gay reader will greet Joy Two and Joy Three with such gratitude, but they will buy them. Weeks before their official release, the Oscar Wilde Memorial Bookshop in New York, which sells only gay books, reported that sale os the two books were brisk. It's said that the first crates were broken open on the side-walk in front of the store, which is on Christopher Street, and that it created somewhat of a sensation.

Aside from the salubrious effects, these books may have on some gay readers, publishers are watching for the "crossover" phenomenon - sometimes with mixed feelings. "We've been told," says Behrnman, "there is somewhat of a market among straight men for the lesbian book." "Setting aside of the male, pornographic mind," says Joy of Lesbian Sex co-author Bertha Harris, "the book will sell to straight as well as lesbian women. There's a market for anything that further moves the veil from female sexuality." She fears, however, that some women might be reluctant to ask for the book and that many won't be able to afford it. (It costs $12.95).

Harris also edits books for Daughters Press, which published her novel Lover and featured Rubyfruit Jungle on its first list in 1972. Daughters sold Rubyfruit to Bantam for $250,000 this year but for some time it has been running in the black on its own, a notable achievement for a small, feminist house which vows to keep all its authors in print.

The small presses - gay, lesbian and feminist alike - are in fact, one of the primary sources for the recent spate of homosexually oriented titles. Many of the bestselling books at the Oscar Wilde Bookshop are books like After You're Out, a collection of gay liberation essays originally published by Links Books and recently re-issued by Harcourt Brace Jovanovich. Half-a-dozen "sex and consciousness" manuals from small houses like San Francisco's Gay Sunshine Press preceded the Joy books. These are books that have already proven themselves with a readership Kent Carroll at Grove Press says is "just as discriminating as the straight readership, perhaps more so." And Jim Landis says it is "a better market than almost any you can hope to find - if you seek to separate it from the general market at all," which assuredly many publishers are trying to do now. Michael Seidman looks forward to the day when gay books "will establish themselves as a genre like science fiction."

There is no doubt that the gay market is still a specialized and geographically isolated market, mainly confined to California, the East Coast and a few points in between, but perhaps not so specialized as publishers might have thought.

In August, McGraw-Hill sent Charles Silverstein, author of A Family Affair, a book for parents of gays, on a cross-country promotional tour which they report was a resounding success - "Anita Bryant didn't hurt us.The only resistance we encountered," says Alice Acheson, "was from the papers - editors who said they had done so much in this field already that they were saturated." The ability of establishment houses to target the market through advertizing in the proliferating gay press and spread their sales via mass distribution at the same time will undoubtedly define the readership potential of what is often called a "hidden minority" of some 11 million.

One might have thought, since the gay liberation movement modeled itself so closely on the civil rights and feminist movements, that there would now be a higher incidence of politically oriented gay books, but what is surprising is the success of light fiction and relatively lightweight nonfiction titles. A few notable exceptions aside, most bestselling gay books, when they treat political issues at all, focus on gay consciousnessness, and not on gay activisim. This is due partly to the special quality of gay identity, that gay oppression is often more internalized than overt. But there are indications as well that gay liberation is somewhat a casualty of the '70s. The current antigay backlash (along with the cut-off of federal payments for abortions and the Bakke case, which charges reverse discrimination in affirmative action programs) indicates that Americans are resisting many of the legislative and social changes of the '60s. More interesting is the almost unavoidable conclusion that gay people themselves are weary of the tensions that have resulted from recent social conflicts. Like all readers, they seek relaxation and amusement from the books they buy, an escape from some of the realities they have helped to define: that they are, indeed, different in many ways from their friends and neighbors and that those differences deserve recognition.