The Republican-controlled Senate yesterday buried the "social issue" agenda of New Right conservatives for the 97th Congress by killing the last remaining matter on the list: school prayer legislation.

The defeat was the second jolting setback in eight days for the conservatives, and brought an end to an 18-month attempt to enact major anti-abortion and school prayer legislation without any measure becoming law.

Sen. Jesse Helms (R-N.C.), the beleaguered leader of New Right forces, said he will drop efforts to enact social legislation until next year.

"I've had my shot at it," Helms told reporters. "I did the best I could."

The defeat of his proposals to strip the federal courts of their jurisdiction on the prayer issue came on three votes. The defeat ended a month-long filibuster against school prayer and anti-abortion legislation, which Helms had attached as riders to a debt ceiling bill.

The first vote, and one seen critical in identifying senators who opposed Helms and the New Right, was to break the filibuster. It failed, 53 to 45, with 60 votes needed for cloture [roll-call chart on Page A5]. The second vote was procedural, which passed 51 to 48.

The third, to send the measure back to the Finance Committee stripped of amendments, was 79 to 16 against Helms.

The second vote, a tabling motion against a Helms attempt to save face and recycle his measure through committee, was considered by many to be the key.

After a series of parliamentary moves, the Senate voted to raise the national debt ceiling for one year to $1.29 trillion from its current "temporary ceiling" of $1.143 trillion by a 50-to-41 margin. The bill, one of the major roadblocks keeping Congress from adjourning, now goes to the White House.

Helms laid part of the blame for his defeat on the White House, although he refused to criticize by name President Reagan, who had lobbied for the anti-abortion legislation and a school prayer constitutional amendment. "I don't know of one single vote the White House obtained for us on the school prayer or abortion issue," Helms said.

Setting up the key second vote, Sen. Barry Goldwater (R-Ariz.) moved to table and, in effect, kill, an attempt to send the prayer amendment to the finance committee with instructions to send it back to the floor. It narrowly passed, 51 to 48, with Senate Majority Leader Howard H. Baker Jr. (R-Tenn.) casting the 51st vote.

Baker, under intense pressure to move the Senate on to other business, said he supported the school prayer initiative but voted to kill it because "at some point this has to end, and this is that point."

A host of other senators then threatened to tie up the Senate with additional amendments to the debt bill. Minority Leader Robert C. Byrd Jr. (D-W.Va.) said Democrats wanted to attach a jobs and unemployment compensation measure. Sen. William L. Armstrong (R-Colo.) said he had an amendment to investigate the use of "slave labor" in the Soviet Union. Sen. Dan Quayle (R-Ind.) offered one to cut the sugar price support program.

When Quayle's amendment was beaten back easily, other senators dropped their proposals and the debt ceiling bill quickly passed.

The debt bill is needed to enable the government to keep borrowing money to stay in business beyond Oct. 1

Helms' school prayer amendment would have kept federal courts from overturning state laws allowing voluntary prayer in public schools, which the Supreme Court banned in 1962.

Opponents called it a "court-stripping bill," saying it would deprive the judiciary of its constitutional powers. Helms said this was nonsense.

Anti-abortion legislation and the liberal filibuster against the prayer legislation have tied up the Senate since Aug. 16, with interruptions only for the Labor Day recess and "must-pass" legislation.

The drive for major new anti-abortion legislation, lobbied for by President Reagan, collapsed Sept. 15. Congress is still expected to pass legislation that would prohibit the federal funding of abortions except in cases where the woman's life was in danger.

At times during the filibuster the Senate floor sounded like a den of lions. Yesterday it appeared to be inhabited by lambs.

Senator after senator praised one another for their courtesy, conviction and hard-fought battle they had waged. Helms said he had never wanted "to move the Senate into hostility." Sen. Lowell P. Weicker Jr. (R-Conn.), a filibuster leader, said, "There has not been one cross word between the senator Helms and me." He told Helms he expects their battles to continue: "Otherwise, you and I wouldn't have anything to look forward to."

The school prayer measure had been expected to pass the Senate easily. Opponents attributed its defeat to a realization of the dangers of stripping authority from federal courts.

"Let this debate be a signal they the New Right cannot end-run the Constitution," said Sen. Max Baucus (D-Mont.), a filibuster leader. "Let this be a signal that there will always be people willing to stand up and fight for the Constitution."

After the vote Helms said he had made a tactical error by allowing the Senate to put off debate on abortion and school prayer until the closing days of Congress. The repeated cloture votes, he said, showed he had support to pass the amendment, but not to break the filibuster.

Helms said his conservative supporters will now use the recorded votes to campaign against senators who voted against him. He insisted that he will again push for the same issues next year. "You win some, you lose some and some days you're rained out," he said. "It will be fair weather another day."

Virginia Sens. Harry F. Byrd Jr. (Ind.) and John W. Warner (R) voted for cloture, the legislative term for limiting debate, and against the Goldwater tabling motion. Maryland Sens. Paul S. Sarbanes (D) and Charles McC. Mathias Jr. (R) opposed the Virginians both times. Sarbanes, Byrd and Warner voted against final passage of the debt limit bill. Mathias voted for it.