Democratic gains in the House and post-election Republican soul-searching in the Senate have virtually eliminated chances of significant action by Congress when it returns Nov. 29 for a lame-duck session that few of its members wanted in the first place.

Prospects for major legislation, already dim before Tuesday's elections, are even dimmer now, congressional sources said yesterday.

The session, demanded by President Reagan during the campaign to prod the 97th Congress into completing its budget work, is likely to wind up without accomplishing much more than passage of a few appropriations bills and more stopgap funding for the rest of the government.

It is not even certain that a defense appropriations bill, considered a priority item by the administration as well as many congressional leaders, will be adopted before the 97th Congress fades into history shortly before Christmas, aides to several leaders said yesterday.

The modest agenda contrasts with a relatively ambitious set of pre-election marching orders from Reagan, including passage of a balanced-budget constitutional amendment, regulatory reform, "enterprise zones" to help revitalize deteriorating neighborhoods, and a new clean air bill as well as appropriations bills.

Even the White House appeared to lower its sights yesterday, however, as deputy press secretary Larry Speakes said that appropriations bills would be the main purpose of the session. Congress so far has passed only three of 13 appropriations bills for the current fiscal year. Other senior White House aides said they now expect little to be accomplished during the lame-duck session.

Especially for the Democratic-controlled House, there is little incentive for action on major bills before the 98th Congress takes office in early January, when Democratic ranks will increase by 26 members as a result of the elections.

But there also seems to be little enthusiasm for a big lame-duck effort in the Republican Senate, even though the GOP will return in January with the same 54-to-46 advantage that it has now.

With House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) looking forward to a 26-vote boost in Democratic strength next year, "Tip won't want to do a darn thing other than a few appropriations bills" in the lame-duck session, House Minority Leader Robert H. Michel (R-Ill.) observed yesterday.

An aide to O'Neill did not dispute the point, although he said that an immigration bill, regulatory reform and job-protection legislation, along with some kind of an anti-recession effort, are "candidates" for action.

"Lame-duck sessions serve pre-election needs, not post-election needs," said the aide, referring to the headlines Reagan had received for summoning Congress to finish its work after the elections.

In the Senate, Finance Committee Chairman Robert J. Dole (R-Kan.) also said yesterday he doubted that the lame-duck session would produce anything more than a few appropriations bills.

But "that's not without precedent," he observed, acknowledging the poor record of most lame-duck sessions in producing major legislation.

Another well-placed Senate Republican source suggested that even the Senate, despite the fact that 95 of its 100 members will be returning next year, needs some time to digest the election results.

"The mood is such that they need more time to decide what the hell they're going to do . . . about defense and a lot of other things," he said.

Noting that even many Republican senators are saying that the administration's defense buildup must be trimmed to help curb budget deficits, this source said he thought the administration might have second thoughts about quick action on the defense money bill.

"If I were Defense Secretary Caspar W. Weinberger, the last thing I'd want to do is bring up the defense spending bill in this climate," he said.

O'Neill has ruled out House consideration of Social Security financing changes in the lame-duck session, and Senate Majority Leader Howard H. Baker Jr. (R-Tenn.) said Wednesday that he would not ask the Senate to go it alone on the controversial issue before the end of the year.

Although the House will not fill its leadership positions for the 98th Congress until January, the Senate is scheduled to fill its posts Dec. 2.

Neither Baker nor Minority Leader Robert C. Byrd (D-W.Va.) is expected to be challenged, although a fight is expected between Sens. Bob Packwood (R-Ore.) and Richard G. Lugar (R-Ind.) for the chairmanship of the National Republican Senatorial Committee, currently headed by Packwood.