President Carter shattered Sen. Edward M. Kennedy's last slim hope of an upset victory tonight and Kennedy withdrew from the fight for the Democratic presidential nomination -- without a public pledge of support for Carter's candidacy.

Kennedy's nine-month effort to derail the incumbent came to a dramatic and sudden end in Madison Square Garden and the Waldorf Astoria Hotel.

After an hour of debate and a 40-minute roll call on the opening night of the Democratic convention, the Carter forces rammed through a rule locking delegates to their original candidate pledges.

The 1,936.4-to-1,390.6 roll-call vote mocked Kennedy's claim that he was within 50 votes of breaking the convention open, and sealed the certainty of Carter's renomination Wednesday night.

But it raised some fresh doubts among the delegates about the value of the prize Carter was winning. The convention hall emptied of Kennedy delegates after the key vote, and even those Carterities who remained seemed dispirited and uncertain about the president's prospects in November.

Ninety minutes after the roll call, Kennedy came before supporters at his convention hotel to say he had called Carter to offer congratulations on "an impressive victory," and to tell his stunned backers: "The effort for nomination is over and my name will be placed in nomination."

Carter sources characterized the telephone call as "very positive" in tone and said the two men had agreed that members of their senior staffs would meet Tuesday morning to seek agreement on their remaining policy differences.

But Paul Kirk, Kennedy's top campaign aide, said about midnight that he had told three senior Carter officials who called him in succession that the Kennedy forces were not yet ready to parley.

"There are no negotiations," Kirk said. "We told them the ball is in their court, and we're ready to work it out on the floor" -- implying there would be platform test votes Tuesday night.

Calling himself "a realist" about his failed candidacy, Kennedy said he would go before the convention Tuesday night to urge support for economic planks that would make it "a truly Democratic platform."

He said nothing about staying over to appear with Carter in a gesture of party unity when the president accepts renomination on Thursday, but Carter forces and neutral party leaders said they hoped that would be the case.

Vice President Mondale, convention chairman Thomas P. (Tip) O'Neill Jr. and Carter campaign chief Robert S. strauss said they believed Kennedy would campaign actively for Carter against Republican nominee Ronald Reagan and independent candidate John B. Anderson.

O'Neill, the House speaker and a longtime friend of the Kennedy family, said he would be "surprised and bitterly disappointed" if Kennedy did not join in the drive.

The need for such unity was emphasized by convention keynote speaker Morris K. Udall, who -- in a departure from the bravado traditional for that role -- said Democrats had only "a fighting chance" of defeating Reagan.

Udall, a liberal congressman from Arizona, and Mondale, who comes from the same wing of the party, held out hope that Kennedy and his backers would swallow their disappointment and work to defeat what Mondale called "a classic right-wing conservative Republican' like Reagan.

"The few differences that divide Sen. Kennedy and President Carter are nothing compared to the vast differences between the Democratic Party and Ronald Reagan," Mondale said.

Udall, a Kennedy backer, inserted an impromptu tribute to the senator at the beginning of his keynote speech, delivered an hour after Kennedy's withdrawal.

But the convention hall was half-empty as Udall spoke and much of the spirit seemed to have evaporated as quickly as the Carter-Kennedy fight had.

Carter aides were unanimous in praising Kennedy's action and saying that it paved the way for a more unified convention and campaign.There were predictions that the Carter side would go a long way toward compromising with Kennedy on the remaining platform issues in meetings Tuesday morning.

But there was also uncertainty about how the 1,200 delegates who came here to vote for Kennedy would feel about being left with only Carter to cheer -- or even whether most of them would return to the convention hall Wednesday and Thursday night to share in the president's triumph.

Two New York Kennedy delegates said they would vote for Kennedy Wednesday despite his withdrawal, and two others said they would probably go home before the voting.

In the bar of the Essex House, four Kennedy delegates from Michigan agreed that the senator's decision was the right one for him. "I don't blame him," said Bill Polakowski of Detroit, who said he thought Kennedy should join Carter on the podium Thursday night only if the president's camp makes more platform concessions.

But Beatrice Williams of Flint said that Kennedy needed to appear with Carter "in the interests of party unity. If [George] Bush could do it with Reagan," she said, "surely Kennedy could do it with Carter."

At another hotel, filled with Kennedy supporters from Calfornia, the mood was bleaker, with several of them saying that the president was a sure loser in their state in November.

Roland Hachey of Maine, a Kennedy alternate, called the senator's decision "a wise political move," but added that he felt as though "my best friend died."

Despite Kennedy's withdrawal, there was no celebrating among Carter's top aides, who said they did not know what additional concessions the senator might ask for if they could accommodate them.

Hamilton Jordan said, "I have every hope . . . that he [Kennedy] will be active in campaigning" for Carter but said the differences between the two camps "may or may not be resolved."

The main issue remaining was the Kennedy-sponsored platform plank calling for a $12 billion federal antirecession jobs program, which comes before the convention Tuesday night.

Before this evening's dramatic developments, Carter White House and campaign aides vowed to fight that plank. But after Kennedy's announcement and what White House press secretary Jody Powell characterized as "a very positive conversation" between Kennedy and Carter, the two camps began fresh discussions on possible compromises.

According to Powell, "they agreed that the task before them was to get this party toegether" and "they agreed that the staff people should get together."

Kennedy's withdrawal overshadowed the debate and vote on the "open" convention rule, which had been the focus of the senator's rhetoric ever since spring, when it became clear that Carter would have a majority among the delegates elected in the primaries and caucuses.

Just as they had used the rule fight -- which would have freed the delegates from their candidate pledges -- to prolong a contest they had seemingly lost weeks ago, the Kennedy forces stretched the suspense of the roll call just as long as they could.

On instruction from the Kennedy trailer, big-state delegations withheld their votes until the end of the roll call. When those votes started coming in, however, it became clear that Kennedy would fall far short of what he needed.

In California, seven Carter supporters deserted the president; in Illinois, 10. But those numbers were far too small to overcome Carter's 700-vote pledge majority, and Pennsylvania's announcement sealed Carter's victory and set the stage for Kennedy's withdrawal.

The rhetoric of the debate recaptured some of the bitterness of the long Carter-Kennedy struggle.

Williams and Ribicoff set the tone for the debate, which for the next 45 minutes repeated all the arguments that had marked the "open" convention dispute in the weeks leading up to the convention.

Rejection of the rule, Mayor Dianne Feinstein of San Francisco said, would deny the 19 million voters in the Democratic primaries their rightful voice at the convention. She charged that the Kennedy forces were "the losing team," demanding extra innings after losing the primaries and caucuses.

The issue, replied New York Gov. Hugh L. Carey, is not Kennedy and Carter but the future of the party and whether the New York convention would resemble "a central party conference" in a totalitarian country of would be "a typical Democratic convention in which unity is achieved through debate."

"You're not cattle, you're delegates and you have rights," Carey said.

The final speaker for the Kennedy side was Sen. George McGovern, the Democrats' losing presidential nominee in 1972, who warned the convention of the consequences of disunity and said an open convention would enhance the party's chances for victory in November.

McGovern said he was grateful, eight years after his own late-night acceptance speech, for the chance to speak in "prime time." Facing a tough Senate reelection campaign this year, McGovern took full advantage of the time slot, ending his speech with an expression of concern for the voters of South Dakota.

Atlanta Mayor Maynard Jackson closd the debate by reiterating the "fair play" argument for binding the delegates to their pledges. Anything else, he said, would lead back to the "backroom dealing" of the old convention days.

The debate over the so-called open-convention dispute began at precisely 6:30 p.m., as agreed to earlier by the Kennedy and Carter campaigns.

Edward Bennett Williams, the prominent Washington lawyer who headed a committee urging an open convention, invoked one of the president's favorite 1976 campaign themes in callng for rejection of the rule binding delegates to vote for the candidate they were elected to support.

"I say to you, Mr. President, trust us, trust this convention, trust the delegates who have pledged to vote for you," Williams said. "Be confident, and let your delegates be free. Let them vote for you in freedom and not in compulsion."

Sen. Abraham A. Ribicoff (D-Conn.), leading off for the Carter forces supporting the loyalty rule, said that both the candidates had competed through the primaries with the understanding that delegate commitments were binding and "it isn't fair to change the rules now."

"It isn't fair to the 19 million Democrats who voted," he said, "and it isn't fair to the candidate who won the most delegates."

While the Carter delegates listened in relative silence to Williams' speech, there was a constant cacaphony of cheers and boos from the opposing camps during Ribicoff's speech.

When Ribicoff said he could support either man as nominee, "but there is a principle involved here," the Kennedy delegates shouted back: "Open it up! Open it up!"

The demonstrators cut short Ribicoff's speech.

Even before the critical rules vote, the focus of attention in both campaigns began to shift to what lies ahead in the remaining platform disputes and the effort to force a public reconciliation between the president and his challenger.

Carter campaign aides were increasingly optimistic that the convention would end with such a show of unity. Kennedy did nothing to discourage them.

Kennedy told the Illinois delegation, which is dominated by Carter supporters, "The only way that the Democratic party can be successful is to reunite it. We have worked in the past together . . . I look forward to working in the future with all of you."

To the New York delegates, Kennedy said that with the right kind of platform and "a nominee that truly believes and is committed to those principles, we will not fail in November."

Kennedy was in friendly territory at the New York meeting, a crowded, noisy and often boisterous affair where Gov. Carey stirred the Kennedy delegates with a plea to reject the "faithful delegate" rule.

"Open it up. Open it up. Open it up," they chanted.

But at the Carter campaign headquarters, the optimism of the president's strategists was reflected by campaign manager Tim Kraft.

After "all the sound and fury" of the debate, Kraft predicted, the dispute over the delegate rule will be "the quickest political disappearing act of the American political calendar of 1980."

"There's no bitterness here," he added.

With the rules fight disposed of, the major point of contention between the Kennedy and Carter forces remained the platform, particularly the economic sections on which the Massachusetts senator is to address the convention Tuesday night.

And on these issues, Carter campaign fficials made clear they had gone as far as they feel they can in attempting to accommodate Kennedy's more liberal views.

On Sunday, the president's strategists announced they had accepted four of Kennedy's proposed platform amendments as they stepped up their drive for a unified convention. Today they said they remained willing to talk to their Kennedy counterparts about the platform issues still dividing them.

But, said White House domestic policy adviser Stuart Eizenstat, "we are not in a position to take any additional unilateral actions" to resolve the platform disputes.

Powell was more blunt. "Lord have mercy, we gave him four things" Sunday, he said.

"President Carter is not going to abandon what we think are well-thought-out positions on the economy just to win a few votes here in New York," top Carter adviser Hamilton Jordan told South Carolina delegates. "We're willing to talk, but we're not willing to make unilateral concessions." "

The main platform debate Tuesday night will center on two Kennedy minority reports that call for a $12 billion economic stimulus program. Eizenstat said this proposal was far too specific for inclusion in what is supposed to be a statement of party principles, and would "reignite" inflation if implemented.

While the Kennedy and Carter campaigns prepared for the platform clash, other groups were pressing their cases on the economic issues.

A group of about 20 mayors, most of whom support the president, met with White House officials to urge inclusion of an antirecession economic package in the platform. The AFL-CIO, meanwhile, lobbied both campaigns to strengthen proposals for stimulating the economy and reducing unemployment.

But in another sign of the apparently growing sense of unity, United Auto Workers President Douglas A. Fraser, a Kennedy supporter, said the remaining economic differences between the two sides were "a matter of emphasis, not principle."

Predicting "a coming together" by Thursday night, Fraser said his main concern was that the party "leave this convention in a united manner."

Both campaigns today announced the speakers who will debate the economic planks Tuesday night. The Carter speakers will include former ambassador to the United Nations Andrew Young, Transportation Secretary Neil Goldschmidt, and Glenn Watts president of the Communications Workers of America. Kennedy will be joined on the platform by Maine Gov. Joseph Brennan; Carol Bellamy, the president of the New York City Council, and Georgia state Sen. Julian Bond.

As the convention was called to order this afternoon and the Carter-Kennedy race neared its final hours, there were some stirrings from late entries in the contest.

Rep. Ronald Dellums (D-Calif.) announced that he would seek to have his name placed in nomination and asked the convention's 481 black delegates to support him.

"I'm suggesting that the blacks at this convention take off their Kennedy buttons, take off their Carter buttons and recognize that they are black," he said.

New Mexico Lt. Gov. Roberto Mondrago, a Hispanic, offered himself as a candidate, and there were stirrings among groups that are unhappy with Carter to find additional candidates as a token protest.

But from the beginning the contest has been between Kennedy and Carter, and with the president remaining at Camp David, the Massachusetts senator enjoyed a day at center stage in New York.

Kennedy, after a morning fund raiser with his sister-in-law, Jacqueline Kennedy Onassis, at the chic nightspot "21," set off on a round of speeches before state delegations around the city.

For the most part, he continued to predict victory both on the rules fight and on the nomination. But he made few references to Carter, and they were mild compared to some of his previous tough rhetoric about the president's economic policies. Further, he told every delegation that he hoped to work in a unified Democratic Party in the future.

He also repeated his assertion that platform concessions alone will not be enough to guarantee his support for Carter if the president is renominated; Kennedy said the nominee must also demonstrate a "substantial commitment" to the party's "fundamental values."

Kennedy was asked how he planned to jdge the sincerity of Carter's commitment. "When a baseball umpire up in Yankee Stadium sees a pitch coming," Kennedy said, "he calls whether it's a ball or strike." Similarly, he said, he would have to decide whether Carter's commitment was adequate. "We'll make the judgment as we hear 'em and see 'em," he said.

The president is to arrive at the convention Wednesday morning and speak to his supporters before the roll call for the nomination that night. In a bit of symbolism growing out of the long Democratic primary battle, Carter plans to land at the nearby Newark airport, where he will be greeted by one of his strongest supporters, New Jersey Gov. Brendan T. Byrne.

Normally, Carter would be expected to land at New York's John F. Kennedy International airport and be greeted by Carey. But the New York governor, while officially neutral in the Kennedy-Carter contest, is known to have urged Kennedy to challenge the president and in recent weeks supported the call for an "open" convention.