It started out as the most sought-after invitation in town, with only a select few summoned to the mountain-top. Then, gradually, the guest list grew well over 100. Now the stay-at-homes are almost more conspicuous than the chosen.

Are the people President Carter has invited to his Camp David summit - what one snubbed senator calls "the last chance saloon" - feeling particularly morose or unpopular? Not on your life.

In fact, some say they wouldn't go if he asked them.

"He saved me the trouble of declining," sanpped consumer advocate Ralph Nader yesterday. Nader has been keeping up with the summit guest list as it unfolds and found himself in good company. "No one in consumer groups has been invited."

"It is not the invitation of the century," snifted Sen. Lowell Weicker Jr. (R-Conn.), one of the nine members on the 18-member Energy and Natural Resources Committe who weren't asked. "I'm not slighted at all."

From Sen. Robert Dole (R-Kan.), ranking Republican on the Finance Committee: "If you're up there, I suppose your status would go up if only from the altitude. The only thing I wondered was if Carter would get down before Skylab, but he missed it."

And from Ohio's Sen. Howard Metzenbaum, one of the handful of Democrats on the Energy and Natural Resources Committee who weren't summoned: "If I had a list of 100 of my priorities of places I'd like to visit, I can't say that Camp David is one of them . . . I'm sleeping pretty well."

Sen. Paul Tsongas (D-Mass.), another member of the Senate energy committee who stayed at home: "There are 535 people in Congress who consider themselves essential to be listened to. Clearly, if everyone who felt he was critical to the nation's survival were at Camp David, they'd have to make a dual-lane highway going out there."

Sen. Howard Baker (R-Tenn.) had a much simpler answer: "He's president and I'm not."

For those who went, however, the summit brought a glossy new image in the eyes of staff, colleagues and children. Wives, of course, thought it only fitting.

Rep. Paul Simon (D-Ill.), chairman of the House Budget subcommittee on inflation, said he and his wife were watching the news Saturday night showing the governors arriving at Camp David.

"Jean said 'You ought to be up there' - every wife thinks that - and five minutes later the phone rang. It was the White House inviting me up for Monday afternoon."

For Simon and other congressional leaders it turned out to be a day with a casual, tieless president who received guests in the elegant, wood-paneled conference room at Laurel Lodge. At a large oak table with a view of the Catoctin woods and ravines, the visitors voiced their opinions at the president's request.

Carter took notes with a blue felt-tip pen and on a couple of occasions Rosalynn Carter joined the group. Accompanying her in the morning was Mary Staggers, who had driven to Camp David from West Virginia with her husband, Democratic Rep. Harley Staggers.

At noon, there were soft drinks and shrimp salad for the visiting congressmen, at dinner, roast beef, salad and chocolate sundaes.

"It was more than just wine at dinner," said Paul Simon. "There's a different rule at Camp David than at the White House."

The mealtime blessing is unchanged. Sen. Mark Hatfield (R-Ore.) told of being caught with his mouth full when Carter asked him to pray.

"I felt a little embarrassed, but I swallowed fast. Then good old Dale Bumpers [D-Ark.] said later that the last time the president asked him to pray, Ted Kennedy turned to him and said it was the worst prayer he'd ever heard.

"And Bennett Johnson [D-La.] misunderstood, thinking that when I said 'Almighty Father' I was referring to the president. He told me he thought I was going a little too far."

Some members of congress felt no compunction about filling in everybody back home on their pilgrimage to the Maryland Mecca. But others outside of government were cautious.

"Everybody was asked not to share the details and I want to honor that," said Claire Randall, head of the National Council of Churches, who wouldn't discuss the menu given her and nine other religious leaders Tuesday night.

Carter's own pastor from Washington's First Baptist Church, Dr. Charles Trentham, wasn't among them. "No one was there except for the heads of national religious groups," he said. I prefer to stay on a more personal level with him."

Also excluded, besides Trentham and 200 million Americans, were:

George Meany, president of the AFL-CIO - "The White House knew well he was recuperating from a knee injury," insisted an AFL-CIO spokesman in explaining Meany's replacement, Lane Kirkland.

Alan Greenspan, former head of the Council of Economic Advisers under Gerald Ford - "I didn't expect to be invited. I'm of the other party."

Robert Lipshutz, counsel to President Carter - "I didn't even give it a second thought. I feel perfectly normal, really."

Eugene McCarthy, former U.S. senator who wasn't invited to the 1976 presidential debates, either - "None of us leaders has been included. All the lobbyists are still in town so I don't think it's a very important meeting."

Bob dole said if Carter had really wanted to make a bold bipartisan move, he would have asked all the Republican candidates up to his mountaintop to give their views.

"That would have been the end of the meeting." CAPTION: Picture, no caption