The Senate handed President Carter a beating yesterday in the first test vote on public financing of Senate elections, refusing by an 11-vote margin to shut off a Republican filibuster. Additional attempts to stop the filibuster are scheduled for Monday and Tuesday.

Republican Leader Howard H. Baker Jr. (Tenn.), who had made extraordinary party-unity pleas to Republicans to oppose the debate cutoff, called the public financing measure "a lousy bill" aimed at helping Democrats win elections.

He predicted on the basis of yesterday's vote that his troops will beable to talk it to death. He said that Carter's "arm-twisting" for the bill, through telephone calls to senators and a public statement in favor of the measure at his Thursday press conference, would avail him nothing.

However, Senate Democratic Whip Alan Cranston (Calif.) and the bill's chief sponsor, Dick Clark (D-Iowa), said the Senate will come much closer on Monday to the 60 votes needed for debate-limiting cloture, and will do good prospect of getting cloture," said Cranston.

Clark said, "I think we're going to win it. Monday we will pick up about five votes and then on Tuesday we will be between 57 and 60. We should make it by Wednesday - maybe Tuesday."

Both strongly denied that the bill would help Democrats more than Repulicans. "Clearly, the bill is going to benefit the Republican Party in the South," said Clark.

Yesterday's vote on cloture was 49 in favor to 45 against, 11 votes short of the 60 required to stop debate, James B. Pearson (Kan.) and Charles McC. Mathias Jr. (Md.) were the only Republicans who voted for cloture; nine Southern Democrats, two Northerners and Harry F. Byrd (Ind. Va.) joined with the Republicans in voting against it. Democratic supporters will need an extraordinary number of switches next week to reach 60.

The bill would apply only to general elections, not primaries. It would provide that the candidate of a major party could receive up to 62.5 per cent of his or her campaign costs from the Treasury, provided the candidate agreed to a spending limit based on state population. The limit would be $250,000 plus 10 cents for each person of voting age, which currently works out to $532.000 for Maryland, $600,000 for Virginia and 1,688,000 for California, for example. A candidate could opt to decline public financing and raise funds fron private sources.

Clark, Edward M. Kennedy (D-Mass.) and spokesmen for Common Cause, the organization working hardest for the measure, contend that public financing for most of a candidate's campaign expenses would free candidates of obligations to special interests that now finance their campaigns, thus removing an increasingly corrupting influence from legislative actions.

However, Baker, Robert W. Packwood (R-Ore.), John Tower (R-Tex.) and other GOP spokesmen are convinced that the bill would help Democrats more than Republicans and is a grossly partisan measure designed to ensure Democratic predominance in the Senate forever. Baker has put heavy pressure on Republicans to stand firm against cloture.

Many Southern Democrats, on the other hand, who now run in what amounts to one-party states, oppose the measure because they fear the availability of public money for potential GOP opponents.

Yesterday, even some GOP sponsors or early supporters such as Jacob K. Javits (R-N.Y.), Clifford P. Case (R-N.J.), Richard S. Schweiker (R-Pa.) and H. John Heinz III (R-Pa.) voted against cloture.

Baker said this was because he had made strong party-unity pleas in which he argued that it wasn't merely a question of this bill, but of showing the Democrats that they cannot ignore the Republicans and do anything they want. He said he had told these and other senators that the Democrats "think if they can roll us on this, they can roll us on anything."

Cranston predicted, however, that the pro-cloture forces eventually would pick up Case, Edward W. B. Brooke (R-Mass.), Charles H. Percy (R-Ill.) and other Republicans, and Clark said he believes Heinz and Javits also would switch to a pro-cloture position once the bow to GOP unity had been made.

Common Cause said a majority of the Senate favors the bill, that it is not a partisan measure and that Baker is using party unity as an "obstructionist tactic."

Republicans believe the bill would help Democrats because it would help incumbents - and there are many more Democratic incumbents. They argue that it would add guaranteed financing to people who already have an advantage because they are known from being in office and having done favors, and because they have a staff and other perquisites of office. Clark argued, however, that the bill could help challengers more by assuring them of at least a minimum of financing which they now lack, especially Republicans in the South.