An effort to end a three-week Senate filibuster against conservative social issues fell apart yesterday when the Senate was told that President Reagan will press for passage this year of a constitutional amendment to reinstate school prayer.

This abruptly halted the Republican leadership's attempts to bring to a vote last night a "must-pass" bill to raise the debt ceiling.

"It appears there is no good and useful purpose to ask the Senate to stay late," Majority Leader Howard H. Baker Jr. (R-Tenn.) said. "I must say reluctantly we will not finish the bill tonight. As a matter of fact, I don't know when we'll finish the bill."

Baker was informed yesterday of the president's renewed interest in the long-stalled school-prayer amendment and his desire for a lame-duck session of Congress after the Nov. 2 elections. This came one day after the Senate handed Reagan and New Right conservatives a jolting defeat on the abortion issue.

As soon as word of this spread on Capitol Hill, "the wind went out of the sails" on efforts to end the filibuster, one congressional source said.

Baker had been trying to negotiate a parliamentary maneuver to give conservatives a face-saving but meaningless victory on the prayer issue. Under his proposal liberals would have ended their filibuster and the Senate then could have voted to reinstate prayer in public schools.

If the measure had been approved, as expected, a motion would have been made to recommit the entire debt-ceiling bill to the Senate Finance Committee, which would have stripped off the prayer language and the full Senate would have passed the bill. The debt-ceiling legislation must be approved by Oct. 1 for the government to borrow money to continue operating.

This would have given conservatives a symbolic victory for the upcoming elections but would not have changed public policy. Thus, liberals could have claimed that their filibuster had succeeded. The deal fell apart when Baker couldn't guarantee Sen. Lowell P. Weicker Jr. (R-Conn.) that the bizarre parliamentary manuever would end the school-prayer debate for the year.

Weicker, a leader of the filibuster, said he had no objection to giving conservatives a "symbolic vic- tory . . . . I'm only interested in realities. They can have all the symbolism they want." Weicker and Sen. Max Baucus (D-Mont.) continued the filibuster, speaking for more than two hours each. Both condemned the prayer amendment, sponsored by Sen. Jesse Helms (R-N.C.), as a "court-stripping" bill. The measure would prevent the Supreme Court from prohibiting states from reinstating prayer in public schools, which the court has done since 1962.

Reagan repeatedly has supported returning prayer to public schools, but his insistence on a vote on a constitutional amendment came as a surprise. The administration drafted the amendment last spring, but the Senate Judiciary Committee has held only three days of hearings on the measure, and it has attracted little public attention.

Congressional sources saw the president's move as a political one to regain support from conservatives embarrassed by Wednesday's collapse of the drive to enact major new anti-abortion legislation. Reagan entered that drive last week.

Some conservatives were unimpressed with the president's efforts.

"The White House didn't lean hard enough," one conservative Senate aide said. "All they made was courtesy calls. It was all for show."

Spokesmen for anti-abortion groups yesterday called their defeat Wednesday a moral victory. In approving a motion to table anti-abortion legislation, 47 to 46, the Senate went on record on the issue for the first time this year, they noted.

"There are now 47 villains out there for us to fight against," said John Mackie, editor of Lifeletter, a respected anti-abortion newsletter. "We now know who they are. That was what this was all about -- getting a recorded vote."