The drive to pass a major anti-abortion bill this year collapsed in the Republican Senate yesterday in a jolting defeat for New Right conservatives and President Reagan, who had endorsed and lobbied for the legislation.

First the Senate refused for a third time, by 10 votes, to break a liberal filibuster against the bill. Then by a single vote, 47 to 46, it voted to table and thus kill it.

The beaten bill had been sponsored by Sen. Jesse Helms (R-N.C.), who suggested afterward that those who had voted against him would face retribution at the polls. "This sets the stage for next year," he said.

Earlier in the day Sen. Orrin G. Hatch (R-Utah) had withdrawn for lack of votes a companion proposal, a constitutional amendment that would have permitted Congress and each state to restrict abortion.

The abortion votes came as House and Senate Republican leaders warned Reagan that Congress, in its rush to adjourn, is likely to brush aside many other items on his election-year legislative wish list, including school prayer, crime and tuition tax credit bills and a balanced-budget amendment to the Constitution. Details on Page A6.

They also agreed there may well be a lame-duck session of Congress after the Nov. 2 election.

In the Senate Finance Committee, meanwhile, the tuition tax credit bill continued to face opposition for civil rights reasons. Details on Page A16.

The Senate floor votes ended an 18-month effort by conservatives to impose new restrictions on abortion, an emotional and highly charged issue. Helms had tried to attach the anti-abortion provisions to a debt ceiling bill that must be passed by Oct. 1 to keep the government operating.

Liberals were ecstatic at the results. "We just weren't going to get rolled over," said Sen. Lowell P. Weicker Jr. (R-Conn.), who led the fight against the Helms bill. "We just weren't going to get snowed under by a moralistic crusade."

A related Helms proposal, which would permit states to allow prayer in public schools, was unaffected by yesterday's action. Weicker pledged to filibuster against it, too, despite an announcement by Senate Majority Leader Howard H. Baker Jr. (R-Tenn.) that he intends to have a bill raising the debt ceiling to $1.29 trillion passed by late today.

Conservatives had felt they had their best chance in years of winning on the abortion and other social issues in this conservative Congress. Liberals had agreed. Yesterday's votes were thus doubly upsets.

Conservatives promised to continue their fight next year, but there are indications that the next Congress may be less receptive.

In the cloture test yesterday Helms picked up only four more votes than he had on the second vote Monday. Helms needed 60 votes to stop debate; he got only 50, to 44 on the other side.

Sen. S.I. Hayakawa (R-Calif.), a conservative, immediately offered the tabling motion. Baker waited until balloting was almost complete and the tally tied before he cast his vote. Instead of voting against the motion, which would have defeated it, he announced he was forming a "pair" with Sen. Robert T. Stafford (R-Vt.) who was absent. If present, Stafford would have favored the tabling move and Baker would have opposed it, he said.

Sen. Jim Sasser (D-Tenn.), who had previously voted with anti-abortion forces, then cast the deciding vote against them.

Hatch, before withdrawing his constitutional amendment earlier, said he had won assurances from Baker that it would be brought to the Senate floor next spring for a full debate. Hatch's so-called "human life federalism" amendment states that "the right to abortion is not secured by the Constitution" and gives concurrent powers to states and Congress to "restrict and prohibit abortion."

Hatch had been promised his amendment would be brought to the floor before Congress adjourned, but he said that he had concluded it would be filibustered.

"If I get into a filibuster, I want to have time to break that filibuster," he told reporters. "I don't want to do it in the last pressure-packed weeks of the session."

Hatch's amendment had been controversial even with some anti-abortion groups. Helms' proposal had been fashioned to draw these groups together. It would have permanently prohibited use of federal funds for abortions and abortion research and set up machinery to expedite Supreme Court review of its 1973 decision legalizing abortion.

The proposal declares the court "erred" in that decision, and that human life begins at conception.

Reagan, who campaigned in favor of many of the conservative social issues, entered into the abortion debate last week, endorsing Helms' proposal and lobbying to break the filibuster.

But when anti-abortion leaders met in the White House with Reagan Tuesday, they found the president had grown pessimistic about chances of success. In telephoning senators, they said the president had found many feared that even if they could invoke cloture the Senate would face additional filibusters that could drag on for days.

Maryland's two senators, Democrat Paul S. Sarbanes and Republican Charles McC. Mathias Jr., voted against limiting debate; Virginia Sens. Harry F. Byrd Jr. (Ind.) and John W. Warner (R) voted for it. Byrd, Sarbanes, and Mathias voted to table Helms' anti-abortion measure; Warner voted not to.