The Senate refused by a 10-vote margin yesterday to break a filibuster against legislation to reinstate prayer in public schools.

A second attempt to choke off the filibuster is to be made today with a third scheduled if necessary Wednesday.

The vote yesterday -- which was 50 to 39, with 60 required -- left both Sen. Jesse Helms (R-N.C.), author of the prayer measure, and filibuster leaders uncertain what the outcome may be as Congress rushes toward adjournment.

"It's very, very close. We're in a one- or two-vote situation," said Sen. Lowell P. Weicker Jr. (R-Conn.), a filibuster leader. "We're in very tough turf."

Liberals used a filibuster to defeat a related Helms-sponsored anti-abortion proposal last week, but his school prayer amendment--which would keep federal courts from acting on state laws permitting voluntary prayer in the schools -- is thought to have more support.

Helms, who has pressed for months for debate on the so-called conservative social issues, yesterday began to express exasperation with liberal tactics and came close to conceding that his school prayer initiative may fail.

After losing the filibuster vote, he complained bitterly on the Senate floor about how liberals have characterized the issue as a constitutional one. "What it is is a matter of whose ox is being gored," he declared.

Later, in a session with reporters, he was far more pessimistic about the chances of success on school prayer than ever before, saying, "I don't know if we'll win."

He complained that Senate Majority Leader Howard H. Baker Jr. (R-Tenn.) has allowed liberals to wage a "Cadillac filibuster" where they've only had to spend two or three hours at a time on the Senate floor. He also vented frustration at the way senators are avoiding up-and-down votes on highly charged emotional issues.

"I've been concerned about 'the club' protecting senators from showing the public where they stand," he said.

The vote yesterday was on a cloture motion to limit further debate to 100 hours. Virginia Sens. Harry F. Byrd Jr. (Ind.) and John W. Warner (R) voted for cloture; Maryland Sens. Paul S. Sarbanes (D) and Charles McC. Mathias Jr. (R) voted against it.

Helms' proposal is attached to a debt ceiling bill that must pass Congress by Oct. 1 for the government to stay in business. It would prohibit the Supreme Court from overturning state laws reinstating prayer in public schools, which the court has prohibited since 1962.

Attorney General William French Smith, the American Bar Association and other legal experts have said the so-called court-stripping bill may be unconstitutional, an encroachment of the legislative branch on the authority of the judiciary. President Reagan has come out in favor of a constitutional amendment to reinstate voluntary prayer in the schools; the White House has been noncommittal about the Helms proposal over the last few weeks.

"The issue is not school prayer. The issue is the Constitution," Sen. Daniel Patrick Moynihan (D-N.Y.) told the Senate. "This is a court-stripping bill. If we pass this legislation, there is no limit to the legislative tyranny that will follow."

A host of Jewish and Protestant religious organizations, the American Civil Liberties Union, the National Education Association and Common Cause, the self-styled citizens lobby, rallied against the measure.

Charles V. Bergstrom, of the Lutheran Council in the USA, said the measure "trivialized prayer" and would jeopardize the rights of religious minorities.

Common Cause president Fred Wertheimer said the Helms measure was "as dangerous a proposal as any considered by Congress in recent memory." He said conservatives were pushing it for political rather than religious reasons.

"This is a last-gasp shot by the New Right in this Congress to leave their mark," Wertheimer added. "When all is said and done, the mark they leave will be failure."