President Reagan faces a new Congress that will be more assertive than the old and will give him fewer dollars for defense while demanding more for job and other social programs, leaders of both parties said yesterday.

Republicans, shaken by several close calls in the Senate as well as their loss of 26 seats in the House, spoke out as strongly as Democrats in promoting new anti-recession efforts, along with restraint in defense spending.

Leaders of both parties joined Reagan in putting out feelers for bipartisan cooperation, suggesting that the Tuesday elections had so wrenched relations between the White House and Congress that extraordinary measures might be necessary to avoid legislative stalemate.

"We don't want anyone to eat crow," said House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.). "The country is in too tough a shape for anything of this nature. I think there has to be some bending on both parts, the party of the Democrats and the party of the Republicans."

"He Reagan will no longer be able to run a partisan government; he will have to run a coalition government," said Rep. Tony Coelho (D-Calif.), Democratic Congressional Campaign Committee chairman.

But some of the most revealing signs of where the new 98th Congress may be headed came from Reagan's own most adept and successful spear-carriers from the 97th Congress, including Senate Majority Leader Howard H. Baker Jr. (R-Tenn.).

Baker said in an interview that Congress will insist on spending less for defense than Reagan wants and "almost certainly" will demand some kind of a jobs program.

"There certainly will be a major effort to trim defense spending and it will be cut more than the administration wants," said Baker, adding that military spending will probably wind up with "something less" than the 7 percent annual after-inflation increase that Reagan has proposed.

"I think it's also almost certain there will be a jobs program of some sort, and there should be," said Baker in an assertion more normally heard from Democrats than Republicans.

Significantly, the idea of a jobs program was also endorsed by Senate Budget Committee Chairman Pete V. Domenici (R-N.M.), who is usually looking for ways to cut spending. Domenici suggested a public works program that would accelerate highway and other projects already under way.

"The American people want to see some signs of progress on unemployment," said Domenici. "Unemployment levels cannot stay at their current levels if we're going to stay on course."

In the House, an aide to Minority Leader Robert H. Michel (R-Ill.), who was very nearly a victim of a recession backlash in the Tuesday elections, said Michel also believes modifications must be made in the Reagan economic program, possibly including less for defense and "some response on unemployment if conditions don't improve soon."

"Perhaps a reduction in defense spending might be one of the areas in which the dialogue opens up between the president and us," said Rep. Guy Vander Jagt (R-Mich.), chairman of the National Republican Congressional Committee.

The potential for conflict over defense and domestic priorities arises from the fact that, before the elections, the administration indicated that it would ask Congress for more defense spending than the congressional budget projects for the next few years,along with major new cuts in domestic spending.

In terms of support for Reagan in Congress, the election took its toll in both houses -- most visibly in the House, more subtly in the Senate.

For most of the first two years of his administration, Reagan got nearly all he wanted from a coalition of Republicans and conservative Democrats in the nominally Democratic House.

But he won many of the critical votes on taxes and spending by only a few votes. The Republican loss of 26 House seats will make that conservative coalition difficult if not impossible to reassemble on most issues, leaders of both parties agreed.

Rep. Barber B. Conable Jr. (R-N.Y.), ranking Republican on the Ways and Means Committee, said Reagan might be able to put it together again to fend off any tampering with his 1983 tax cut. But others said the old coalition would be beyond reach on most other issues.

"The coalitions of the past are not the coalitions of the future," said the Michel aide. The minority leader is "looking at a more pragmatic solution . . . at more compromise to avoid confrontation and stalemate," the aide added.

The elections "might have been a message to alter the course, not change the course," said Vander Jagt, putting a fine distinction on the meaning of Reagan's campaign plea to "stay the course."

In the Senate, where voters kept the 54-to-46 margin for Republicans but gave half a dozen incumbents good scares in close races, the membership next year will be only slightly less conservative than it was this year. The real question is the election's impact on this year's survivors and those who will be facing reelection in 1984 and 1986, several party leaders said.

Moderates who barely won reelection this year "are going to be more difficult to hold in line," along with the 19 Republicans who will be up for reelection in two years, said a key GOP source who asked not to be identified.

"Senator Baker is going to have a more difficult time holding the troops together for some type of programs," he added.

"It augurs for a quite different Senate," Sen. Bill Bradley (D-N.J) said at a press conference on the election's impact. "The moderate Republicans just elected and more moderate Republicans who are up in 1984 . . . will not go in lockstep" with conservative policies.

"Clearly the slogan, 'It's Not Fair, It's Republican,' was telling," said Sen. David F. Durenberger (R-Minn.), a moderate who won Tuesday after a hard fight and who contends that defense will have to bear a greater share of any new budget cuts in the 98th Congress.

An aide to another moderate Republican who barely squeaked through on Tuesday said: "There's no question that they the moderates will continue to be very independent. Their independence is why they're coming back. And the guys who are up in 1984 and 1986, what are they going to do?"

Echoing O'Neill from the House, National Republican Senatorial Committee Chairman Bob Packwood (R-Ore.) said, "there's going to have to be compromises . . . to the extent the president wants to get part of his program, he's going to have to give."

Packwood said that, despite stay-the-course comments from the White House on military buildup and other issues, he thought Reagan would wind up compromising with Congress, as he did with the California legislature when he was governor and as he did last year with Congress on tax increases and spending cuts.

Others were not so sure. "We're not sure they've gotten the message," said one GOP congressional aide. "They'll get it," said another, grimly.

On Social Security, the message may be that the elections so politicized the whole issue that effective action to curb costs may be impossible, Baker suggested. The campaign debate over the future of retirement benefits "scared people to death -- I've never seen anything like it," Baker said.

Social Security will not be taken up during the lame-duck session starting Nov. 29, and "I'm not at all certain we can ever muster a majority of brave men and women to do anything" in a major way on the issue, he said.