The Vatican has directed five American theologians to recant what it calls doctrinal "errors" in a controversial book that challenges church rules on sexual behavior.

The Vatican's Congregation for the Doctrine of the Faith, which in an earlier day banned books and excommunicated their authors, directed that the five theologians "give rigorous reconsideration" to the position that they have assumed in their 1977 work: "Human Sexuality: New Directions in American Catholic Thought."

The book challenged traditional church doctrine by suggesting that in certain circumstances, such forbidden actions as homosexuality, birth control and premarital sex may not be considered sinful.

Church scholars have agreed that no American church officials in modern times have been the subject of such censure from the church's top watchdog agency on faith and doctrine. Since the reforms of the Second Vatican Council of 15 years ago, the Index of Forbidden Books has lapsed into history, authors no longer are required to secure the imprimatur of a bishop certifying the orthodoxy of their work and only rarely does the Vatican publicly inveigh against a work or an author.

The high-level criticism of the American theologians is of added significance since the work in question was commissioned by the prestigious Catholic Theological Society of America.

The basic thrust of the book was to discard the "thou-shalt-nots" of the church's traditional rules of sexual behavior and instead establish guidelines to help the faithful make their own decisions about right and wrong in the light of firm moral principles applied to each situation.

Using such criteria the authors - two priests, two laymen and a nun - suggested that in certain circumstances, actions traditionally forbidden by the church, such as contraception, premarital sex and homosexual acts, might not automatically be considered sinful.

The theologians set as prime criteria for the morality of any sexual act the question of "whether specific secual behavior realizes certain vales that are conducive to creative growth and integration of the human person." They also set forth seven principles for evaluating sexual behavior: "Self-liberating...other-enriching...honest...faithful...socially responsible, life-serving...joyous". The "substantial violation" of any of these values, in any sexual act, they wrote, "should raise serious question about the ability of that sexual expression to enhance creative and integrative growth of the human person."

The Vatican Congregation is sharply critical of such an approach because it offers "no manageable or ehelpful rules for serious conscience formation in matters of sexuality." In the book, moreover, they are called "guidelines" which can never be regarded as "absolute and universal moral norms," the Vatican statement said.

The Vatican statement charged that in applying their criteria to specific sexual questions, the theologians "either dissociate themselves from or directly contradict Catholic teaching as consistently proposed by moral theologians and as taught by the church's magisterium."

Traditional Catholic teaching, the Vatican statement pointed out, holds that "the use of the sexual function has its true meaning and moral rectitude only in true marriage."

The Vatican statement was transmitted by Cardinal Franjo Seper, who heads the Congregation for the Doctrine of the Faith, to Archbishop John R. Quinn of San Francisco, president of the hierarchy in this country, and by the hierarchy's headquarters here to all U.S. bishops, but apparently not to the five theologians involved.

Yesterday - three days after the U.S. bishops' press office had sent copies of the statement to reporters in this country - the Rev. Anthony Kosnik, the editor of the controversial book, said he had not yet seen a copy of the statement.

The Vatican statement, which is titled "Observations of the Congregation for the Doctrine of teh Faith", mentions no sanctions if the theologians decline the "reconsideration" called for.

Three of the five teach in Catholic seminaries. Kosnick declined to speculate whether the high level reprimand might affect their tenure. "Until they tell me to pack my bags, I'm staying here," said Kosnick, dean of Sts. Cyril and Mthodius Seminary in Rochard Lake, Mich.

The other authors are Sister Agnes Cunningham of St. Mary of the Lake Seminary, Mundelein, Ill.; the Rev. Ronald Modras, St. John's Seminary, Plymouth, 7ich.; James Schulte, St. Joseph's Hospital School of Nursing, Marshfield, Wis.; and William Carroll, John Marshall School of Law, Chicago.

The original hardback version of the book sold 30,000 copies, according to a spokesman for his publisher, Paulist Press. Doubleday is planning to bring out a paperback version later this year.