President Reagan pasted his 49-state reelection landslide into his scrapbook yesterday, but even before the glue set critics began challenging his claim to a renewed mandate for his conservative policies.

Reagan told a Los Angeles news conference that by giving him a 59-to-41 percent majority over Democratic challenger Walter F. Mondale, one of the greatest routs of the modern era, "the people made it very plain that they approved of what we're doing."

But Democratic leaders said that while Reagan won a record 525 electoral votes, the voters had blurred the message by trimming two seats from the GOP majority in the Senate and cutting off Reagan's coattails well short of the point that would have restored the Republican-Southern Democratic conservative coalition control of the House.

"There is no mandate out there," said an obviously relieved House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.). O'Neill, sounding more like the manager of a winning fighter than a trainer whose contender had just suffered a standing knockout, made it plain that the 70-seat Democratic majority in that body would force Reagan to take the political heat for any painful measures to trim budget deficits and would fight him on further cuts in social spending.

In a surprising rebuke to the newly reelected head of his party, House Minority Leader Robert H. Michel (R-Ill.) told the Associated Press that Reagan was to blame for the disappointing GOP gains in the House because "he really never, in my opinion, joined that issue of what it really means to have the numbers in the House."

Michel said Reagan seemed more eager to win an unprecedented personal 50-state sweep than to help GOP congressional candidates, and he cited a last-minute presidential stop in Mondale's home state of Minnesota, which ultimately eluded him by a narrow 15,000-vote margin, as an example of that focus. Speaking of Reagan, Michel said, "Here the son of a buck ended up with 59 percent and you bring in only 15 seats" in the House.

Talk of Reagan's second-term victory heralding a political realignment almost disappeared yesterday as White House officials and leaders of both parties focused on the coming battle for supremacy on Capitol Hill between the powerful but lame-duck president and what seemed yesterday to be an unabashed Democratic opposition.

Democrats said their two-seat Senate gain -- the product of takeovers in Illinois, Iowa and Tennessee and an unexpected loss in Kentucky -- weakened the leverage of the GOP majority, now 53 to 47, and promised a restoration of Democratic control in 1986.

And they contended that the new, 251-to-181 Democratic majority in the House, with three seats still unsettled, was cohesive and self-confident enough to block at least some of the expected Reagan efforts at budget cutting.

Reagan told reporters that he was prepared to "take our case to the people" if the incoming Congress balks, as he did in pushing his initial tax and budget cutting and defense spending plans through in the first eight months of 1981. Aides noted that Reagan had run up huge margins in many Democratic House districts, as he powered his way to one of the great electoral triumphs in U.S. history.

Reagan won more than 70 percent of the votes in four states, more than 60 percent in 28, and more than 55 percent in 11.

While flexing his political muscle, White House officials also held out the prospect of cooperation with the Democrats, saying they hope to get Democrats and Republicans together on a single bipartisan tax simplification bill and push it through Congress.

Rep. Jack Kemp (R-N.Y.), a proponent of the GOP version of that measure, said he already had been in touch with Rep. Richard A. Gephardt (D-Mo.), principal House Democratic sponsor of a similar plan, and was hopeful of "working out a compromise the White House could approve."

Kemp and his fellow conservative, Rep. Newt Gingrich (R-Ga.), blamed what they called the "disappointing" House GOP gains not on any lack of effort by Reagan but on the persisting weakness of the grass-roots Republican Party.

"Reagan did his part," Gingrich said, by appearing with embattled Republican candidates in targeted campaign stops during the final two weeks of the race. "We didn't have the ability to reach out for him from the grass-roots."

It was clear from post-election comments that Reagan's role had been a vital one in races other than his own. In Kentucky, the only state where an incumbent Democratic senator was unseated, for example, Reagan piled up a margin of 281,000 votes, while Jefferson County Judge Mitch McConnell (R) beat Sen. Walter D. Huddleston (D) by 3,300 votes.

Last-minute presidential visits and Reagan coattails could not save Sens. Charles H. Percy (R-Ill.), who lost to Rep. Paul Simon (D-Ill.), or Roger W. Jepsen (R-Iowa), defeated by Rep. Tom Harkin (D-Iowa). The third Democratic pickup came in Tennessee where Rep. Albert Gore Jr. (D-Tenn.) took over the seat of retiring Senate Majority Leader Howard H. Baker Jr. (R-Tenn.).

The gubernatorial elections, as usual, were on their own path with Republicans gaining four states, North Carolina, Rhode Island, Utah and West Virginia, and the Democrats taking over three, North Dakota, Vermont and Washington.

But the biggest GOP breakthroughs in the congressional battles came in North Carolina and Texas, where Reagan's 62 and 64 percent margins helped save two embattled Republican Senate seats and provided coattails for seven House pickups, three in North Carolina and four in Texas.

Nonetheless, the size of the overall GOP gains -- though very much in line with historical precedents for presidents elected to second terms and with private White House preelection predictions -- fell far enough short of the publicized hopes of congressional Republican leaders as to mute the Republicans' celebration of Reagan's own victory.

White House chief of staff James A. Baker III said, "It was a victory for his philosophy and a victory for him personally, but I'm not sitting here claiming it's a big mandate. I'm simply saying that the size of the mandate is going to be determined by how successful we are in the next four years in getting the president's programs and policies through the Congress."

But even that fairly modest claim was rejected by some Democratic leaders on Capitol Hill, who were talking as if they had won.

"We stopped the realignment in the House in 1982," said Rep. Tony Coelho (D-Calif.), the chairman of the Democratic Congressional Campaign Committee, "and we kept it stopped in 1984."

There was a tone of defiance coming from many Democrats, including defeated candidate Mondale, and almost a sense of glee at the problems they said Reagan soon will face.

Before returning to Washington, D.C., and starting a vacation, Mondale said that the Reagan administration will have to digest "some of the biggest meals of crow" ever devoured, because of what he called unrealistic promises of cutting budget deficits without raising taxes.

"I think the American people are going to be very angry," when Reagan inevitably reneges, he said.

In a similarly skeptical vein, O'Neill said, "We are going to be very fair with the president. He made a tremendous amount of promises and pledges to the American people. We are going to give him the opportunity to keep them."

Coelho underlined the tone of defiance by saying that his message to the reelected president was: "As of today, you are a lame duck. Accept it. Elected officials do not have you to contend with any more."

While Democrats were crowing defiance, Reagan's campaign director, Edward J. Rollins, said, "The real message that's out there is the old-style liberalism of the Humphrey-Mondale-Johnson era has sort of been repudiated by the American people for a second time."

Republicans did not claim that this automatically would make them the majority party but said that Reagan's big margins among the youngest of voters gives the GOP its best opportunity in recent history to expand its base.

"The lasting legacy President Reagan has left to the future is those 4 or 5 million potential new Republican voters," Rollins said.

Acknowledging the GOP's appeal, former Mississippi governor William Winter, who was buried in his bid to unseat Sen. Thad Cochran (R-Miss.), said, "The Democratic Party does have to take a long look at itself and build along the lines that will attract more of the younger voters of this country. This is where I see the Republican Party as having made serious inroads -- in attracting people under 30, people entering business and the professions."

Sen. Gary Hart (D-Colo.), whose support among younger voters almost gave him the Democratic nomination in the fight with Mondale, said in a statement that the party needs to broaden its base in recognition of the nation's changing demographics and appeal to voters in the South and West as well as the East and Midwest.

"We must learn from our losses if we are not to repeat them," Hart said, perhaps with his own eye on 1988. "We can no longer simply propose redistribution of wealth; we must also create more opportunity for everyone, including the middle class."

Sen. John Glenn (D-Ohio), another contender for the 1984 Democratic presidential nomination, said, "The first thing we've got to realize is that this loss is not just an aberration for the Democratic Party. This is four out of the last five presidential elections that we've lost."

To Glenn, that means the party must realize the huge numbers of Democrats who were at the poverty level or below it "when I was a kid growing up in the Depression" have moved up to middle-class status and "we have to address their concerns."

Georgia Democratic chairman Bert Lance and other southerners agreed wholeheartedly. Lance, who was pushed out as general chairman of the Mondale campaign after his appointment drew a storm of protest, said too few elected Democrats were willing to campaign for the national ticket because of a sense that wasn't where the voters wanted to go, especially in light of Mondale's call for higher taxes.

That same view was espoused by a number of Democratic governors, who said they wanted to assert a stronger role in reshaping the party.

Texas Gov. Mark White said, "The national party needs to be restructured to be more reflective of the people of this country, and the governors are in a position to play a vital role."

Kansas Gov. John Carlin said, "We should have learned before this that we have to wake up to the 1980s and stop trying to relive the '50s and '60s."

Other Democrats who echoed the theme that the the national party must appeal more to middle-income America included Sen. Lloyd Bentsen (D-Tex.), the chairman of the Democratic Senatorial Campaign Committee and one of those considered by Mondale as a running mate.

Bentsen said the long and intricate process of winning the Democratic nomination "moved Fritz pretty far over to the left side. He obviously tried to work his way back during the campaign, but he couldn't do it."

"On the state level, the Democratic Party does not have that image," Bentsen added. "And we did all right in the governorships, the Republicans didn't do what they thought they would in the House and we ran absolutely against the tide in the Senate . . . . I think people also like the checks and balances in the system. I think enough of them felt that way to make the difference" in the congressional races.

Lance also argued that the reason Mondale lost so badly was the presence of Geraldine A. Ferraro, another person "perceived to be liberal," on the ticket.

"This has nothing to do with the man-woman issue," Lance said. "We had two southern women who were available -- Louisiana Rep. Lindy Boggs and Kentucky Gov. Martha Layne Collins. It was a question of philosophy. There's a reason for the fact that the idea of a balanced ticket has lasted for 200 years. Some old ideas are pretty good ones. . . . We did not choose someone on the ticket with Mondale who would have been able to attract the kind of people I'm talking about."

Not all Democrats agreed with the view that the party's ideological stance was the main problem. Jesse L. Jackson, who mounted the first serious campaign by a black candidate for the Democratic nomination, blamed Mondale's defeat on the failure of poor whites to recognize their common economic interest with blacks and support the same candidate.

New York Gov. Mario M. Cuomo said Mondale's defeat underlined that "you have to have someone who can campaign effectively" on television, but said it would be "dangerous" for Democrats to accept the "Republican view and endorse the denial of compassion."

AFL-CIO President Lane Kirkland offered no second thoughts for his federation's solid backing of the Democratic ticket.

"We remain convinced of the soundless of our position in the primaries, the caucuses, and the general election," Kirkland said. "Mondale-Ferraro fought for the right issues . . . . The AFL-CIO remains undaunted. We shall continue to work for 'liberty and justice for all.' "

Former senator George McGovern (D-S.D.) took wry comfort in being stripped of his title as the worst loser in a presidential election, "at least in terms of electoral votes." McGovern won only Massachusetts and the District of Columbia in his 1972 race against President Richard M. Nixon.

McGovern said he thought the election reflected "a modest trend toward conservatism" in the country, but not a fundamental shift.

"I think people are fairly content with right about where we're at right now," McGovern said. "I think they feel it's just about the right mix" in Congress and the White House. "We're reviewing public assistance programs, but not really slashing them. We're watching the Russians, but keeping open the possibility of a nuclear freeze and arms control. I think the average American would like the president to negotiate with the Russians and get the deficit under control."