A federal court here has dismissed a lawsuit against the National Conference of Catholic Bishops (Nccb) brought by the Costello Publishing Company of Northport, N.Y.

Long Island publisher Harry J. Costello claimed in a suit filed in 1976 that the NCCB's liturgy committee violated antitrust laws when it took steps to discourage the sale of an unofficial prayer book.

The Costello firm held that its plans to distribute in this country an Irish breviary entitled "Morning and Evening Prayers" were blocked by the NCCB. The publisher asked the court to issue a permanent injunction against the NCCB's actions.

The NCCB said the publisher had failed to submit the book for approval in accordance with the rules of the Catholic Church.

During the long-running dispute, some critics of the bishops had charged that the NCCB was motivated by financial considerations. Royalties from the sales of the NCCB-approved liturgy book go to the International Commission on the English Liturgy, of which the NCCB is a member.

A church spokesman said that for the past two years, the American hierarchy has not shared in profits from prayer-book sales, although it has received grants from the international commission in previous years.

In dismissing the suit, U.S. District Court Judge June L. Green noted the bishops' responsibility for approving liturgical texts used by Roman Catholics in the U.S.

"Concomitantly," she said, "the NCCB has the duty to discourage the dissemination and use of unapproved liturgical texts in the diocess of the United States."

Green noted that the bishops had notified booksellers that the plaintiff's book was not an approved text, and said this cannot be considered the type of activity prohibited by the Sherman (antitrust) Act.

"These actions had no commercially competitive motive, but were undertaken as an exercise of the duty each of the . . . defendants has to protect the integrity of the Roman Catholic liturgy worldwide. Such religiously motivated conduct is not the type which falls within the realm of the antitrust laws."

Judge Green also held that the activities of NCCB and other defendants involve religious activity, and therefore "that the free-exercise clause of the First Amendment prohibits this court's review of ecclesiastical decisions and activities challenged herein."

In a controlling decision by the U.S. Supreme Court, the judge said, "judicial review of ecclesiastical decisions was forbidden even where allegations of religious impropriety were raised. The court notes that no such allegations are presented herein. Thus, this matter presents even stronger prudential considerations."

Wilfred R. Caron, general counsel for the NCCB and for the U.S. C atholic Conference, said that "the right of the bishops to protect the integrity of liturgical texts has been upheld, and an important precedent has been established."

Publisher Costello said his lawyers would consider an appeal after studying the opinion.

"The decision is a blow to American Catholics and publishers," Costello said. He held that his Irish breviary "is acknowledged to be vastly superior" to the NCCB's approved American text.