Sen. Edward M. Kennedy, still trying to prod President Carter into a face-to-face confrontation, said today that he would withdraw from the race for the Democratic presidential nomination if Carter debated him and then won the June 3 round of primaries.

The White House quickly rejected the idea.

Kennedy issued his challenge at the Los Angeles Press Club in a quietly dramatic speech that depicted Carter as a failed president who is virtually certain to lose in November unless he defends his policies in the Democratic primaries.

"His fear of debate speaks louder than any words about his failures in office and his prospects in the general election," Kennedy said.

The Massachusetts senator, in effect, offered a carrot and a stick to his opponent, who leads him nearly 2 to 1 in the battle for delegates to the Democratic National Convention.

The carrot was Kennedy's offer to quit if he lost on June 3, thus giving Carter a chance to heal some of the wounds in his party in preparation for what promises to be bruising fight against Ronald Reagan and John B. Anderson in the general election.

The stick was Kennedy's vow to continue the race if Carter refused to debate him, or if Carter debated him and then lost the majority of the Democratic votes on June 3.

"I know the Democratic Party," Kennedy told a crowd of cheering, hand-picked supporters at the press club. "I know that if a debate is not held in the primaries, it will be heard at the convention. I could not prevent it if I wished to; the delegates to any Democratic convention would require that their nominee stand for something greater than a belief in his own victory. So if President Carter is as concerned as he claims for party unity, let him debate the issues before the primaries end."

On public television tonight, Kennedy said Carter's refusal to debate will subject the president to "a good deal of resentment" from voters in California and other late-primary states.

The president's rejection of Kennedy's challenge came in a statement issued by the White House press office.

"Our position regarding the conduct of the campaign is well known and remains unchanged," the statement said. "We certainly agree with Sen. Kennedy that the June 3 primaries are important. It is essential to keep in mind, however, that this is a political process which began Jan. 21 in Iowa, and that the voters in the vast majority of states have so far indicated a clear and unmistakable preference for President Carter's continued leadership.

"We will remain confident that after June 3 the president will have secured the Democratic nomination by a wide margin."

At a news conference here, Vice President Mondale heaped scorn on his old Senate colleague, calling his offer "a desperate effort" and "a ridiculous proposition."

"I don't see any reason why an incumbent president should debate a loser in his own party," Mondale said.

A Carter campaign official said "there wasn't any question" that the Kennedy proposal would be turned down. Calling the proposal "frivolous" and "one-sided," the official said it appeared to have been made with the expectation that it would be rejected.

"If they had been serious about trying to find a way to reach an accommodation, this is not the way to do it," he said.

Within his own camp, Kennedy's speech was regarded as his most important since he drew the liberal policy lines of his challenge in a Jan. 28 address at Georgetown University.

"It's the speech he should have given last November," said state Comptroller Kenneth Cory, a leading California supporter who introduced Kennedy today.

Kennedy, who has appeared tired and somewhat dispirited on the campaign trail this week, also attached considerable importance to his new tactic. At the last minute he canceled a public appearance in Las Vegas on Wednesday night to give himself added rest. Today he delivered his indictment of Carter's economic and foreign policies in a crisp, low-key tone containing none of the stridency heard in some earlier speeches.

His campaign, said Kennedy, was not only to win the nomination but also to "define the soul of the Democratic Party."

"To those who say that issues do not matter, that they are something to run on before the election and to run from afterwards, let us answer that a political party must stand for things that are more enduring," Kennedy said. "To those who count the political cost of a debate, let us cite the human costs of economic inaction, and social injustice."

As in the past, Kennedy hit hard at Carter, blaming the president for "serious weakness" in the military forces, and calling his economic policy "a pale carbon copy of the Reagan policy." But his central charge was that Carter was short-circuiting the democratic process by his consistent refusal to debate the issues.

Kennedy said he had declared his willingness to debate at the time he led Carter in public opinion polls last November. He said Carter was then "anxious" to debate and was willing to do so for the first 60 days after the capture of the American hostages in Iran.

"What changed his mind was not the situation of the hostages but the shift in the polls," Kennedy said. "His reason was in Iowa, not Iran. Since he now regards the crisis abroad as 'manageable' and since he now feels free to give days of his time to campaigning he can certainly manage to free an hour of his time for a debate."

Kennedy's offer to get out if Carter debated him and won the June 3 primaries is tied solely to the popular vote total, not the number of delegates or the number of states either candidate carries.

Most of the votes will be cast in three big primaries -- California, Ohio and New Jersey. The other states holding primaries on June 3 are Montana, New Mexico, Rhode Island, South Dakota and West Virginia.

A total of 696 delegates will be chosen in these primaries -- more than a fifth of the convention total.

The latest delegate count by United Press International shows Carter with 1,521 delegates, Kennedy with 813 and 86 uncommitted. These are delegates firmly pledged to one candidate or the other or projections of the anticipated outcome in states that have not completed their selections.

Whether Kennedy can win a majority of votes on June 3 with or without a Carter debate is an open question.

Carter is believed to be ahead in Ohio. And in California, where Carter has never been strong, a Los Angeles Times poll this week showed him running five points ahead of Kennedy. Kennedy's prospects in California are further dimmed by the presence on the ballot of Gov. Edmund G. (Jerry) Brown Jr., who has withdrawn from the race but may get some protest votes that would otherwise go to the Massachusetts senator.

But whatever the outcome on June 3 Kennedy's challenge today gave new impact to his oft-repeated desire for debate.

"We've put the ball in Carter's court," said a Kennedy aide. "If he really wants to have party unity this is his chance to get it."