Liberal Roman Catholics, faced with growing signs that their church is determined to make abortion the keystone of efforts to diminish dissent in this country, are divided over how to fight back.

Yesterday, the Committee of Concerned Catholics used a full-page advertisement in The New York Times to protest reprisals it said have been taken against signers of an ad in that newspaper 17 months ago.

But some respected voices on the Catholic left have expressed fears that yesterday's statement may exacerbate the situation.

The controversy grows from the Oct. 7, 1984, ad that disputed the church's view that the only "legitimate Catholic position" on abortion is that it is always morally wrong. The ad said committed Catholics may hold "a diversity of opinions" on abortion and called for discussion.

Coming in the heat of the presidential election campaign, the ad was in part a rebuff to New York Cardinal John J. O'Connor for his widely publicized rebuke of Democratic vice-presidential nominee Geraldine A. Ferraro, who had said she would not allow her religion to dictate her political stands.

Before the year was out, the Vatican office in charge of religious orders told the 24 nuns who were among the 97 signers of the ad to recant or face expulsion.

After more than a year of tense negotiating in secret, nine of the women were cleared, although none recanted.

Within the past four or five years, Vatican efforts to reimpose strict orthodoxy on the vigorous and diverse American branch of the church have multiplied.

Bishops here have been investigated; a popular catechism in use for a decade was suddenly ruled unacceptable; the pope dispatched one committee to scrutinize American theological seminaries and another to carry out a detailed study of men's and women's religious orders in this country.

A high-level group of American bishops, on their official once-in-seven-years visit to the Vatican, was criticized by Pope John Paul II for the "permissiveness" in the American church.

But of all the issues that provoke Vatican chastening, abortion seems to be the flash point.

*A nun, Agnes Mansour, was forced to choose between her continued membership as a Sister of Mercy and a high-level job with the Michigan social welfare department because part of the funding she administers pays for abortions for poor women. She kept her state job and resigned from the order.

*Mary Ann Sorrentino, executive director of Rhode Island Planned Parenthood, was excommunicated last year because that agency arranges abortions. Sorrentino's daughter, before her first communion, was grilled by her priest because of her mother's job.

"Abortion is treated as a kind of litmus test" of orthodoxy, said Rosemary Ruether, a Catholic theologian who teaches at the United Methodist Garrett Evangelical Theological Seminary in Evanston, Ill., and was a signer of the 1984 ad.

Many of those signers have reported reprisals:

*Daniel C. Maguire, professor of theology at Marquette University, had longstanding speaking engagements canceled at four Catholic colleges: St. Martin's College in Lucey, Wash.; St. Scholastica College in Duluth, Minn.; Villanova University and Boston College.

Both Villanova and Boston paid him the stipends he would have earned, prompting the former Jesuit priest to observe that he felt "like the American farmers now being paid not to produce." He has filed an academic freedom complaint with the American Association of University Professors.

*A lecture by Ruether last January was abruptly moved from the Catholic Family Society of Santa Rosa, Calif., which had invited her, to a United Methodist church down the street when the bishop received a protest against Ruether from a militantly antiabortion layman.

*In Los Angeles, the director of Catholic Charities ordered employes not to refer battered women to shelters operated by Sister Judith Vaughn, one of the signers.

*Jane Via, who has degrees in theology and law and who teaches part time at the University of San Diego, a Catholic school, has been told she will not be allowed to speak at any meeting in the diocese until she publicly affirms her agreement with official doctrine on abortion.

*Last summer, the Vatican's official representative in this country, Archbishop Pio Laghi, and San Francisco Archbishop John R. Quinn boycotted a session of the official Leadership Conference of Women Religious -- the umbrella group of all heads of women's religious orders -- because one of the 24 nuns, Sister Margaret Farley of Yale Divinity School, was a featured speaker.

For many liberal Catholics, most of whom share their church's abhorrence of abortion in principle but hold that exceptions are sometimes justified, the dismay at the Vatican's threat to the 24 nuns was matched by uneasiness over aligning themselves with the group behind the first ad, Catholics for a Free Choice.

Meanwhile, the National Catholic Reporter, for two decades the bible of liberal Catholicism, assailed the second ad as "a deceitful, dishonest and divisive effort." In an editorial, the paper said the only beneficiary of the long, bitter dispute was Catholics for a Free Choice, "which reaped a rash of publicity," adding that "It is the women religious whose image has been severely -- even though perhaps unfairly -- tarnished by the ad controversy."