The Oval Office and much of the White House West Wing are empty now as President Carter travels to another summit - but among those Carter loyalists who remain behind, a feeling has set in that is far gloomier than emptiness.

It is despair.

A genuine political despair, perhaps unmatched in any modern White House, except in those very last days of Nixon.

It is a despair that Carter may have been so severely crippled by the latest gasoline crisis - and by a public perception that he is not coping with it - that he may be kept from reelection. And it is a despair, perhaps most unsettling to those Carter loyaltists of midlevel rank, that the problem has grown largely because the president and most of his highest echelon have not realized just how bleak the political reality is.

For months, Hamilton Jordan has taken comfort in the view that while Carter has shown poorly in public opinion polls, he will do much better when people begin to focus on specific matchups: Do they want Jimmy Carter or Jerry Brown as the next president? Carter or Ronald Reagan? Carter or Howard Baker? Jordan has also argued within the inner circle that the public will see that none of these other candidates has answers for such problems as energy or inflation. Jordan's view has been widely repeated as gospel within the inner circle.

"But now," said one midlevel White House official, "there seem to be no assurances that the public will not become so disillusioned that they will want to take a chance on someone else - anyone else - as president, whether he matches up favorably by Carter, issue by issue, or not. I'm afraid there is really an anybody-but-Carter mood growing. And we just don't seem to be doing anything about it.

Carter's slide in the public opinion polls is starting to look more like a plummet. He fell to a new low last month - just above Richard Nixon's low point, according to the latest from Louis Harris. Republicans Reagan and Baker are defeating Carter in one-on-one matchings in the latest polls. Democrat Edward Kennedy, who talks about supporting Carter, continues to defeat the president 2 to 1; Democrat Brown, who talks about challenging Carter, still loses to the president but is within striking distance, trailing by only a dozen or so points.

And, fairly or not, Carter receives overwhelmingly negative ratings for his handling of the energy crisis, as the public places much of the blame on the president for failing to act, not on the Congress for failing to enact.

But what troubles Carter loyaltists the most is that the president went off to his big power summit in Tokyo after paying scant visible attention to the crisis at home.

"It's frustrating," says one White House official. "Here the president was heading off for a week and a half at a time when all hell was breaking loose over gas lines on the East Coast and out in Dallas and Houston and the West."

Events, international and domestic, seem to have been working against the president.

Carter flew back from Vienna earlier last month heady with the signing of the strategic arms limitation treaty and eager to begin the fight for its ratification. But he landed in the midst of an American public that was short of temper and long of gas line frustration - and certainly in no mood to have SALT rubbed in its energy wounds.

Carter's energy secretary, James Schlesinger, had been putting out conflicting signals in the president's absence, saying one day that the oil companies had "unduly" aggravated the crisis, and saying the next that the oil companies could not be blamed.

Carter's top advisers met to discuss what the president should do. He could have gone on television, perhaps with a five-minute fireside chat as Franklin Delano Roosevelt used to do in times of crisis, and as Carter did once a long time ago.

He could have demonstrated that he was taking charge. He could have told the public that he would not permit the oil companies to act against the public interest and that the Justice Department was investigating to insure that the companies were not boosting prices or holding back gasoline (Justice officials are probing this matter, but that is not very well known).

There was discussion of what the president could do between the two summits (Vienna and Tokyo)," said one official. "But no one came up with anything really compelling he could say or do."

So Carter said and did relatively little.

Instead, the Carter officials opted for what one concedes was "a somewhat indirect route." They decided that every day, in at least one way, the White House would put out something to do with energy up until the time Carter departed for Tokyo. This was done.One day it was a statement about the truckers' strike; another day it was explanation that the summit in Tokyo would really be an energy summit, not an economic summit.

They did this daily thing, but, as some Carter officials now concede it was a subtlety that was lost on many of those who had plenty of time to read their newspapers and listen to the news while waiting in gas lines.

"He could have gone on TV and said he was going to call in the oil companies and talk tough with them," one adviser said. "But it's just not in Jimmy Carter's nature to do that until he knows all the facts."

One who has stayed behind in his White House office is the de facto secretary of energy. Not James Schlesinger - he's just one who has the formal title - but Stuart Eizenstat. Eizenstat's the man Carter quietly put in charge of coordinating all energy policies, apparently because he wants someone other than Schlesinger to be running the show, even though he has this thing about not firing Cabinet people. So Schlesinger stays, and Eizenstat, already overburdened as chief of all things domestic, does still one more job.

Assistant Secretary of Energy Al Alm is working for Eizenstat on a task force that will give Carter July 8 what Eizenstat calls "a major study on energy production and conservation."

The study, Eizenstat said in an interview, is an attempt to look at energy alternatives that have been much discussed in recent years - solar, increased use of coal, synthetic fuels, unconventional gas from tar sand, for example - as well as the future of Mexican oil. The study will determine how much each alternative will cost, compared to its equivalent in conventional oil.

"This has never been done before," Eizenstat said. It is good that it is being done now, but if it hasn't been done in the past, one wonders what Schlesinger's Department of Energy has been doing with its time and the taxpayers' money.

Eizenstat said he expects that Carter will go on television sometime after he gets this energy report and will demonstrate anew his intent to deal with the problem. "That's one reason for this report," Eizenstat said. "We wanted to make sure he had some answers to provide when he got in front of the cameras." Eizenstat added:

"I believe the latest OPEC price increase will prove to be a real watershed event. It will show that we cannot continue to mortgage our country's future to OPEC.

"I hope this will be the stimulus by which the president can mobilized the country to act."

So it is that defacto secretary in the White House office is hoping that OPEC will mobilize the country to act while others in the White House are hoping to mobilize the president's highest echelon to act. Sometimes the White House inner circle is a vicious one.

"No one seems to be taking this energy crisis seriously," says one Carter insider who remained in Washington. He is an adviser of some standing, and is so loyal to Carter that he shakes with frustration, even rage, as he talks.

The Carter inner circle is divided, he says, and unsure of what to do about the energy crisis or how to do it. Among those closest to Carter, only press secretary Jody Powell realizes the full political depth of the crisis. He has been pressing Carter to act, publicly and decisively. Eizenstat, too, realizes the gravity of the situation, but he is dreadfully over burdened.

This adviser says that neither Carter nor Gerald Rafshoon views the problem as one of a great political severity, and that Hamilton Jordan "has gone into hiding this week." (Jordan's secretary says he is not in hiding, it is just that he is not available because he is at the beach.)

One adviser from the outside who is said to preceive the severity of the problem is Carter oldster Patrick Caddell, who is said to have been so concerned that he called Carter in Tokyo, urging that he cancel his planned Hawaii vacation and come home. (Carter decided to do just that).

One way to understand the nature of Carter's political problem is to try to come to grips with what he can do to solve it. And it is here that many of the Carter staff members fall into despair. They look at the significant accomplishments of the Carter presidency: the Camp David accord and the heroic shuttle that brought the unprecedented Israeli-Egyptian peace treaty, civil service reform, the Panama Canal treaties, significant cuts in unemployment, increased appointments of blacks and three years without a single American life lost in millitary combat.

And then they look at what each has gotten Carter: A momentary blip in the polls at best, more often just a brief pause in the steady decline in public opinion.

Carter has come home with a SALT pact and few seem to care. Politicians courting the right wing attack him; politicians courting the middle and the left keep silent. The one person who speaks out the loudest turns out to be Andrei Gromyko, whose idea of help is to perform the most delicate figure of Russian forensic subtlety since Nikita Khrushchev did his table-top tap dance as a way of pounding home his debate points.

So great is the concern among Carter staff members that some are driven to overstatement. "The mood here is like the last days of the Nixon presidency," one middle-echelon Carter aide, who knows the Nixon White House only from the books he has read, tells a friend. "Everyone's ready to throw in the towel. They think the situation is hopeless."

For Carter the situation is not hopeless, but it is not good. To see the scope of Carter's problem, it is only necessary to look at Carter's success in 1979.

He won then by offering himself not as a man with a bold new plan, but as a man who was different at a time when what people wanted most was a change. He offered a different background (on Washington), a different approach (reform and reorganization) and a different style (press conferences in blue jeans, pond drainings in Plains).

In 1976, Carter was a man who offered nothing specific, but something different. In 1979, when placed side by side with Carter in the latest polls, Ronald Reagan is starting to look like a fresh face.