An effort to end a four-day Senate filibuster against anti-abortion legislation fell apart yesterday when liberals refused an offer by Sen. Jesse Helms (R-N.C.) to put the proposition to a quick vote.

Senate sources said that meant the Senate would adjourn for its Labor Day recess without disposing of proposals by Helms on abortion and school prayer, and debate would continue in September.

Senate Majority Leader Howard H. Baker Jr. (R-Tenn.) intends to file a cloture petition today in an attempt to end the filibuster, according to aides. The procedure, which would require the approval of 60 senators, won't be considered until after the recess.

Helms aide James P. Lucier said opponents refused Helms' offer because they lacked the votes to defeat his proposals to restrict access to abortion and keep the Supreme Court from reviewing state laws permitting prayer in public schools.

"They want to kill them without voting," he said. "You don't filibuster if you have the votes."

Helms and Sen. Bob Packwood (R-Ore.), leader of the filibuster, have been under pressure all week to end the abortion-prayer impasse. Helms yesterday said he was ready to vote on anti-abortion and school prayer amendments he has attached to a debt ceiling bill as well as two amendments from liberals designed to gut his proposals.

Packwood, after meeting with allies, refused Helms' offer. He said he offered a counterproposal under which liberals would agree to a "free-standing abortion vote" if it didn't include school prayer and if it weren't attached to the debt ceiling bill. Helms rejected this offer.

"We'd like to focus on abortion," Packwood said. "With school prayer you muddle the issue."

Yesterday's debate was conducted almost entirely by Helms' opponents. They denounced his proposals as unconstitutional efforts to restrict the authority of the Supreme Court.

During the day, liberals gained one conservative ally, Sen. Barry Goldwater (R-Ariz.), who voted for two Helms-sponsored school prayer measures that passed the Senate in 1979. He said he opposes Helms' school prayer amendment "much as I'd like my grandson to pray in school" because it would have school officials write the prayers.

Sen. Lowell P. Weicker Jr. (R-Conn.) later addressed the same issue, saying: "What is a prayer to be? Is it to be a Protestant prayer, a Catholic prayer, a Jewish prayer, a Buddhist prayer . . . or is it to be a mishmash of every religion known to the world and therefore meaningless?"

Helms' two amendments are to a debt ceiling bill that must be passed by Sept. 30 for the government to continue operation. The first would prohibit the Supreme Court from ruling on state laws relating to voluntary prayer in public schools. The high court outlawed public school prayer in 1962.

The second amendment would impose a host of restrictions on abortions, and includes the finding that "scientific evidence demonstrates the life of each human being begins at conception." It declares the Supreme Court "erred" in its 1973 decision guaranteeing women access to abortion in the earlier months of pregnancy, and provides for a direct appeal to the high court on state law restricting abortion.

The Helms proposals would permanently prohibit the use of federal funds and federally funded insurance policies to pay for abortions. Helms' aides said the measure has the support of from 45 to 55 senators, but conceded the vote would be close.

Weicker and Sen. Max Baucus (D-Mont.) have introduced amendments to gut Helms' proposals by reaffirming the authority of federal courts to rule on all constitutional issues.