Despite claims of victory by Sen. Edward M. Kennedy and acknowledgements of some "softness" from the backers of President Carter, the odds last night remained against success for the Kennedy-backed effort to overturn the proposed Democratic National Convention rule binding delegates to their candidate pledges.

Kennedy said the "'open' rule will win by 2 to 1, or 3 to 1," but officials in his campaign acknowledged that the "hard count" of delegates ready to overturn the binding commitment was about 150 votes short of a majority.

Robert S. Strauss, chairman of the Carter campaign, said the "worst case" estimate for the president still gave him a 350-vote margin over the Massachusetts senator on what both sides agree will be the key vote at the convention.

The roll call, scheduled for Monday evening, will be on a Kennedy challenge to the proposed convention rule allowing replacement of any delegate who threatens to bolt from the candidate preference expressed at the time of his or her election.

Carter goes into the convention with 1,985 delegates to Kennedy's 1,243, with 103 uncommitted, according to United Press International. Unless the "loyalty rule" is overturned, Kennedy strategists acknowledge that Carter's renomination is a certainty.

There has been speculation that Carter might release his delegates to "vote their consciences" if he he wins the rules fight, but Strauss said last night that this was "absolutely not" in prospect.

Strauss and others in the Carter organization conceded yesterday that the margin on the rules fight will be substantially closer than the 742-vote edge by which Carter's delegates exceed Kennedy's.

Those familiar with the president's intensive delegate-monitoring operation said that "soft spots" of varying size were showing up in such states as New York, Michigan, Illinois and Pennsylvania. Although one of the Carter head-counters claimed "we have not lost a delegate in the last 72 hours," there was some nervousness as operations shifted from here to New York City, site of the convention.

Nonetheless, there was a general acknowledgement that Kennedy and the late-starting, autonomous "open" forces face formidable obstacles of numbers, organization and political psychology in attempting to break Carter's grip on the convention.

So far as the numbers are concerned, no independent audit of delegate sentiment has yet found nearly enough restlessness among the Carter delegates to suggest that the rule is likely to be overturned.

A Washington Post poll of a sampling of 591 delegates July 22 to 30 found 54 percent for the "loyalty rule," 41 percent against, and 5 percent undecided. Nearly identical percentages were found in an ABC TV poll of 800 delegates released yesterday. A United Press International survey of 25 percent of the Carter delegates in each state found 89 percent of them supporting the president's position on the rule. That is not a sufficient deviation rate to overturn it.

Even if the members turn out to be closer than they now appear, convention officials pointed out, the Carter forces will have important tactical advantages in a floor fight.

Kennedy's allocation of floor passes was increased from 11 to 120 on Tuesday as part of a compromise agreement setting the time and conditions of the rules and platform debate. But the Carter forces, these sources said, will probably have three times as many floor operatives available -- a vital element in controlling any last-minute rebellions.

The communications system on the convention floor is in the hands of pro-Carter officials, it was noted. And the Carter operatives have more recent experience in floor battles, both at the 1976 convention (also held in Madison Square Garden) and in the 1978 miniconvention in Memphis, than Kennedy's forces can bring to bear.

Pro-Carter veterans of past conventions acknowledge that in the excitement and confusion of a floor fight, a mass psychology sometimes develops that sweeps away the fail-safe systems installed by the candidate supposedly in control.

But in the realm of psychology, these officials noted, the "open" convention forces appear to face some internal problems that may slow the kind of bandwagon sentiment they need.

Although Kennedy has been the chief proponent of the "open" convention effort and is a possible beneficiary of its success, some of the others promoting that cause see his continued presence in the race as a barrier, not a help, to its success.

Keith Haller, an aide to Rep. Michael Barnes (D-Md.), a leader of the congressional bloc promoting the "open" convention, said yesterday, "Our problem is that both the Carter and Kennedy people are telling the delegates that if the convention is opened, Kennedy is the alternative."

Haller, a spokesman for the open convention forces headed by Edward Bennett Williams, said, "An amazing number of state chairmen tell us that if Kennedy pulled out, they'd like to move, but the fear of Kennedy keeps them locked to Carter and the rule."

The Williams-Barnes group officially has no preference for the nomination, although the group's principal sponsors are close to Secretary of State Edmund S. Muskie. Formation of a separate committee to promote Muskie for the nomination was announced yesterday, paralleling a similar group set up a few days ago for Sen. Henry M. Jackson (D-Wash.).

Others would like to see Vice President Mondale substituted for Carter, but both Mondale and Muskie have rejected any endorsement of moves on their behalf.

The failure of the open convention forces to agree on a desired alternative to Carter or Kennedy is seen by the Carter forces as another factor limiting the prospects for a successful rebellion.