Something resembling panic swept through Capitol Hill Democrats yesterday as more than three dozen House members, worried about President Carter's floundering reelection chances, met secretly to plot how to obtain an "open" convention.

A statement calling for both Carter and Sen. Edward M. Kennedy (D-Mass.) to release their delegates at next month's Democratic National Convention was circulated in the group. But, according to several of those present, the discussion quickly turned to how to come up with a Democratic presidential candidate other than Carter or Kennedy.

"There was an unmistakable consensus in the room that we need a non-Kennedy, non-Carter alternative on the ticket," said one congressman, who asked not to be identified. "The names of Fritz Mondale and Ed Muskie were mentioned prominently."

While alarm was spreading among Democrats in Congress, there was no evidence of a similar reaction among about 200 Carter delegates gathered yesterday at the White House. Based on interviews, the delegates are holding firm for President Carter.

"In Congress, they're always worried," laughed delegate Richard Austin, a Democratic leader from Detroit. "They were worried last year, you know. That's what got Kennedy into it. But we don't [worry]. That stuff isn't going to affect the nomnation at all."

The members of Congress included supporters of both Carter and Kennedy who came together because "we are all appalled by the prospect of a Ronald Reagan presidency," said Rep. Michael Barnes of Maryland, selected as a spokesman for the group.

But there was also a strong element of self-preservation involved. "Clearly, if the election were held today, Jimmy Carter would be defeated resoundingly," Barnes said, "and a lot of good people in Congress would go down with him . . . There is the real possibility that we're facing a debacle."

This fear among some congressional Democrats was reinforced by a new Louis Harris survey released yesterday that indicates that Republicans lead Democrats by 47 to 43 percent in the race for control of the House.

In March, Democrats held a 50-to-41 percent lead over Republicans. Harris said the results "raise the distince possibility that in 1980 there may be a decisive shift in base party strengths, and that the GOP now has a real chance to win both the White House and control of the Congress."

Reps. Anthony Toby Moffett (Conn.), a Kennedy supporter and Thomas J. Downey (N.Y.), a Carter supporter, organized the two meetings of concerned Democrats. About 15 attended the first early morning session at the Capitol Hill home of Rep. Fortney H. (Pete) Stark Jr. (Calif.). aBy 1 p.m., when the second session was held in the Rayburn House Ofice Building, the group had grown to 35 to 40, according to various sources.

Among those in attendance were Reps. Don Edwards of California, Timothy E. Wirth of Colorado, Patricia Schroeder of Colorado, Jim Weaver of Oregon, Shirley Chisholm of New York, John Conyers Jr. of Michigan, Jerome A. Ambro of New York, Barbara A. Mikulski of Maryland and Ronald V. Dellums of California.

A split developed in the group over the call for an "open convention" after Carter supporters and others who do not favor Kennedy complained that the statement would sound like a pro-Kennedy move. Another meeting is scheduled for Monday to pursue the issue.

"It would be unfortunate if this were perceived as a hidden way to nominate Teddy Kennedy," spokesman Barnes said. "What we want is someone who can unite the party and that everyone could rally around.

"The name we kept hearing was Ed Muskie," he added. "Vice President Mondale came up, and I also heard talk of a Muskie-Scoop [Sen. Henry M.] Jackson ticket, which has a great deal of appeal to me."

The uneasiness over Democratic prospects in November has been growing all week. Some of the concern is linked directly to the Billy Carter affair, but a Harris poll earlier this week also showed Carter trailing the Republican nominee by 28 percentage points.

Northern Virginia Rep. Herbert E. Harris II said he and others are "outraged at the whole Billy Carter thing, at the process by which he gets involved in foreign policy. Some of us react very adversely to this and it raises questions that have to be answered in a public forum. I see potential fallout unless it is met head-on. Those who feel the convention should have more than one choice have been spurred on by this.

"All of us want as strong a presidential ticket as possible," Harris added, "and it isn't very strong right now."

Illinois Rep. Paul Simon, who faces the additional burden this fall of an opponent named John Anderson (not the presidential candidate), said the Billy Carter revelations hastened the mounting alarm in the House.

"I hear it from everyone, and there is no question about it," Simon said. "But a lot can happen between now and November, remember."

Another Democrat with local home-district political problems, Rep. Andrew Jacobs Jr. of Indianapolis, said his fate is tied to Carter's because of a straight-party lever on voting machines in Indiana.

"My ankle is chained to him by that lever," Jacobs said, "and the process for casting other than straight-party votes is difficult."

An open convention with delegates free to vote for any candidate has long been the center of Kennedy's longshot strategy to capture the Democratic nomination. The first and most important Carter-Kennedy fight at the convention will be over a party rule that requires committed delegates to vote for the candidate they were elected to support.

A CBS poll of delegates found a majority for the closed rule, but enough of them are wavering that they might change their minds. This would give Kennedy an outside chance to win the rules fight.

But there is no evidence to date to indicate that Carter's 700-vote delegate lead is evaporating.More than 200 Carter delegates from coast to coast were in Washington yesterday for briefings from various Carter aides and a White House reception with the president and Mrs. Carter.

Carter, in a brief pep talk to the delegates, made no reference to the rumors of a "dump-Carter" effort. The president portrayed the Democratic contest as a race between himself and Kennedy, and held out an olive branch to Kennedy supporters.

"We have strong and good and sincere Democrats who are supporting Sen. Kennedy," he told his backers. "Give them respect and a hand of friendship."

Interviews with many delegates present indicated they are firm in their support of the president, and will vote with Carter forces at the convention, not only for the nomination but also on rules and platform questions.

"Many of these delegates were elected when the president was in tough political shape. They've been with him through thick and thin," said Larry Conrad, Carter's Indiana delegate chairman. "Why should they leave him now?"

Conrad, a former candidate for governor, said every Carter delegate from Indiana was screened in advance. "When you've played political games as long as I have you know your people, Conrad said. "I know my people are solid. If I tell them you've got to topple a church steeple, all they say is, 'what's the best way to do it?' They'll be the same way with Carter."