In whispered conversations on the Senate floor and in the cloakroom, Democratic senators agree there is only one way to get rid of Jimmy Carter in 1980: send a delegation of senior Democrats to the White House to convince him to step down in favor of Ed Muskie.

That such a step is even considered testifies to intense fears of an unbearable Carter drag in the 1980 election, fanned less by the Billy Carter affair than last weekend's Lou Harris poll warning of a Republican takeover of Congress. Yet the Senate talks and worried huddles on the House side agree that, barring new sensations, there is no way to deprive President Carter of renomination if he fights back.

These members of Congress realize what much of the public and many politicians do not: the decade of "reform" has so radically changed the nominating process that the threatening anti-Carter talk the last few days is largely irrelevant. The Democratic National Convention is almost totally removed from the influence of the party's elected officeholders.

Officeholders, many seeking November reelection, are panic-stricken by polls and headlines. A few even fear a Ronald Reagan landslide and a seminal election on the lines of 1932, rearranging party balances.

What's more, they are in remarkable agreement about what to do: replace Carter but not with Sen. Edward Kennedy. Far and away the overwhelming choice is Secretary of State Muskie. If he were paired with a southerner (say, Sen. Sam Nunn from Carter's Georgia), the odds favoring Reagan would become less formidable and officeholders seeking reelection would breathe easier.

As late as 1968, such force of opinion would have threatened the end of Carter. But a dozen years of reform have thinned the ranks of officeholders among the 3,331 delegates.

Nevertheless, anti-Carter forces propose an "upstairs-downstairs" strategy to open up the convention in the Aug. 11 rules fight, leading to the nomination of Muskie (or perhaps Vice President Walter F. Mondale). Members of Congress "upstairs" will call for rebellion by the delegates "downstairs," which will result in a Carter rules fight defeat. But just who these "upstairs" rebel leaders will be is another matter.

Word was quietly spread, for instance, that Sen. J. James Exon of Nebraska might lead the way to open the convention. While a freshman senator and scarcely a national power, former governor Exon is a moderate Democrat well respected in the Senate. But aside from telling us that Carter ought to "show a little strenght" by foregoing the proposed rule binding delegates, Exon avoided taking sides. In fact, he is not attending the convention.

That is true of a surprisingly high number of his colleagues in both houses of Congres who see many reasons not to be in New York in August. Rep. John Brademas of Indiana, the House majority whip wished for by stop-Carter forces as a leading player in the drama, will be nowhere near Madison Square Garden.

But even if Exon, Brademas and others braved summer in New York, their impact on Carter's 1,990 delegates would be minimal. They were picked for loyalty to the president, know few of the endangered office seekers and lose little sleep over the health of the Democratic Party. As Mondale's political aide, Richard Moe, has said, the delegates of 1980 are merely agents of the candidate to whom they are pledged.

Most worried Democrats on Capitol Hill understand this new order. Their only hope is that Carter, following Lyndon Johnson's 1968 example under radically different circumstances, will withdraw. But since Carter will not, talk has turned to sending a congressional delegation down Pennsylvania Avenue to ask the president to step aside for the good of the nation (and the party).

In the Senate, talk has centered on a three-senator delegation: Robert Byrd of West Virginia, the Senate majority leader; Abraham Ribicoff of Connecticut, retiring with honor after 18 years; and Carter's esteemed fellow Georgian, Sam Nunn. All have deep reservations about the president, and none has supported Kennedy.

It would take worse news and gloomier polls to dispatch any such delegation down the Avenue before Aug. 11. But even so, would Carter and submit as President Nixon did when three Republican elders traces a similar route in August 1974?

Nixon was on the brink of impeachment; Carter is nowhere near it. The former governor f Georgia, who emerged from obscurity to become president against the party establishment's wishes and then overcame the odds again to whip a certain part of Teddy Kennedy's anatomy is not about to heed any plea from Senate elders. Jimmy intends to keep those Carterite robots disguised as delegates, whatever frightened Democrats in Congress want.