A homosexual prisoner and four publishers failed a suit yesterday challenging as unconstitutional regulations under which "homosexually oriented" books and magazines are barred from the federal prison system.

In a suit filed in U.S. District Court, the plaintiffs asked that the U.S. Bureau of Prisons be directed to drop its policy concerning such publications.

Exhibits filed with the complaint included an exchange of letters between Rep. Edward I. Koch (D-N.Y.) and Norman A. Carlson, director of the Bureau of Prisons. Koch said two banned publications with which he has became familiar in his study of prisoner rightw were the magazines, It's Time, and The Advocate.

The congressman said both publications "surely must be judged as legitimate." He said any decision "regarding the receipt by prisoners of literature relating to homosexuality (should) be based on reasonable criteria."

Carlson replied that "publications which call attention or identify inmates who accept homosexuality can, in our opinion, be detrimental to their safety as well as to the safety of others."

The suit was fied by the National Gay Task Force, publisher of It's Time; GCN, Inc., publisher of Gay Community News: Off Our Backs, publisher of Off Our Backs; the Inside-Outside Collective, Inc., publisher of Join Hands, and by Calvin Keach, a homosexual prisoner at the federal correctional facility in El Reno, Okla.