President Reagan's working coalition in Congress was thrown into jeopardy last night by a tidal wave of Democratic votes that carried incumbent Republican House members out of office and created a tense and uncertain mood at the White House.

Sen. Robert J. Dole (R-Kan.), the most outspoken member of the GOP leadership team in the Senate, said the administration was "taking a bath in the House" and that this meant that Reagan would "have trouble getting his program through." Later he withdrew the word "bath" and substituted "shower."

A jubilant Martin Franks, executive director of the Democratic congressional committee, said of the president: "He no longer has ideological control in any way, shape or form." House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass) called the congressional elections "a disastrous defeat" for Reagan.

At the White House, chief of staff James A. Baker III tried to put the best face on the outcome of an election that was clearly disappointing to Republicans. Baker said that the president was "upbeat" because of the Republican retention of the Senate. But only a few days ago White House spokesman Larry Speakes, after conferring with Baker, described the notion that the Democrats might take over the Senate as "the stuff of which pipe dreams and political columns are made."

Yet the administration last night refused to concede that its working majority in Congress was gone. "We still think we have the makings of a governing coalition in the House and we're very encouraged by the results in the Senate," said White House communications director David R. Gergen.

Republican losses in the East, followed by strong showings by GOP candidates in the West, created a see-saw mood at the White House. Officials were encouraged by early fragmentary returns, downcast by gubernatorial losses and network projections of heavy House losses in mid-evening and then uplifted somewhat as these projections were scaled back.

The apparent victories of Sen. John Danforth in Missouri and Republican Senate candidate Pete Wilson in California further heartened the administration, and Speakes said shortly after midnight, "It's not looking bad."

As it became clear that Republicans had failed to meet their House expectations, Baker tried to subtly shift the White House pre-election goals. He told a late-night briefing at the White House that a loss of 20 to 30 seats would be considered "a wash."

Last week, in a detailed briefing for The Washington Post, an administration official said the predicted range of GOP losses was from 13 to 27 seats and that anything higher than this would be a setback for the administration.

Baker acknowledged last night that it would be "tougher" to get legislation through the House, assuming a 25-seat loss, "but it will still be possible to have support for the president's program on selected issues."

The White House chief of staff made his comments after delaying his briefing for an hour in the hope that returns from western and southern states would improve the administration's position.

"We do not want to react to network projections," Baker said tersely after returning from a dinner in Reagan's private family quarters with other high-ranking administration officials. "We want to react to results."

As Baker spoke these words at 10 p.m., CBS projected a Republican loss in the House of 33 seats and NBC projected a loss of 25. This approached the upward range of the pre-election "worst case" prediction at the White House of 27 seats.

The administration was especially worried that more than 19 incumbent GOP House members might lose. This is because endangered Republican incumbents were in almost every instance opposed by liberal Democrats who would not become part of the GOP-conservative Democrat coalition in the next Congress.

It was not immediately clear whether the White House had succeeded in this goal. With western and many midwestern returns still to come, House GOP incumbents had lost or were trailing in 16 districts.

Even the loss of 25 GOP seats posed the possibility of an executive-congressional stalemate characteristic of the Ford and Carter years.

Sen. Gary Hart (D-Colo.) gave voice to this fear last night, saying, "We are looking at two years of stalemates . . . . "

Hart added that any talk by the White House of resurrecting the 1981 coalition that approved the original Reagan economic program is "whistling past the graveyard."

It is not only the economic program that is threatened by an ideological shift in the House. Such varied measures as the farm bill and the nuclear freeze initiative were decided by a two-vote margin in the present Congress.

The depth of the Republican losses sent Baker and other top aides into a high-level huddle at the White House following the dinner with the president. As usual, Reagan was nowhere in evidence.

When a reporter asked whether the president was "groaning when he looked at the losses," Baker said only, "The president is not groaning."

Democrats were joyful about the outcome.

Former vice president Walter Mondale said that the results showed the voters "don't want to stay the course." Mondale said they wanted instead "a mid-course correction."

Sen. Edward M. Kennedy (D-Mass.), a leading Democratic prospect for the presidency, said: "I think now this election calls for alteration in our economic policy."

Anne Lewis, political director of the Democratic National Committee, said she was "cautiously optimistic" that there would be a working majority of traditional Democrats in the House.

Earlier in the evening, after Gergen had said on the basis of fragmentary returns that he had a "strong belief" that the administation would keep its working coalition, Lewis commented that the voters "are clearly sending a message to the president that they want a change."

"If the White House is interpreting tonight as a vote for more of the same than we are in for some serious misunderstandings in the next Congress," Lewis said.

There were scraps of consolation for the White House amidst the ruins of defeat last night. Edward J. Rollins, the administration political adviser who is recuperating from a mild stroke at George Washington Hospital, cited the election of Sen. Paul Trible in Virginia and a late surge of Republican candidates in the west.

But Rollins, who had once privately forecast a GOP gubernatorial loss of eight to 11 statehouses, said that the heavy Democratic gains in the governorships "are not good for us."

Gergen took consolation in an NBC exit poll showing that 48 percent of voters thought that Reagan economic policies should be given more time to prove themselves. However, the poll also showed that only 7 percent of the voters believed that the policies were successful while 37 percent regarded them as failing.

Another finding in this survey showed that 48 percent of independents supported the GOP, down only 2 percent from 1980. One administration official called this "the most significant statistic we have seen."

Administration officials are concerned that the president will come out of the election looking badly, particularly because he made the national election something of a referendum on his own policies by his "stay the course" slogan.

But if the exit polls were any indication, many voters found nothing inconsistent in saying that the president's program should be given more time to work and then voting for Democratic candidates who would like to change it.

Reagan's role in the election campaign was a source of dispute among his own aides throughout the year.

The president was heavily involved in fund raising for GOP Senate incumbents, and the GOP used him on radio and television messages in many districts.

But Reagan campaigned in few populous industrial states during the fall campaign. Some candidates, such as the victorious Trible and GOP Senate nominee Pete Wilson in California, made it clear they didn't want the president in their states.

Reagan wound up the campaign following conflicting strategies. He taped a generally worded television message for the final weekend of the campaign. Before that, he conducted a two-day campaign in five sparsely settled western states where it was believed that the president's popularity could make a difference.

The president wound up his campaigning in New Mexico, where a limited survey the next day showed incumbent Republican Sen. Harrison Schmitt pulling ahead of his Democratic challenger, Jeff Bingaman. But Bingaman forged ahead again the next day, and he defeated Schmitt last night.

Republicans had been prepared to claim that Reagan made the difference if Schmitt had won.

Interpreting the election early in the evening yesterday, one administration analyst suggested that the difference between the "frost belt" of the northeast and the "sun belt" of the southwest may have widened.

An ABC exit poll yesterday showed that President Reagan had only a 42 percent approval rating in the northeast compared to 50 percent approval in the west and 54 percent approval in the south.