President Reagan's latest budget initiative suffered a major setback yesterday as House Minority Leader Robert H. Michel (R-Ill.) flatly rejected proposed cuts in Social Security and said he would not push for House passage of Reagan's recent budget "compromise" with Senate Republicans.

"You've got to take that off the table before you even start," Michel said of the $40 billion in unspecified Social Security savings over the next three years that the Senate Budget Committee recommended last week, with Reagan's blessing, as part of a revised budget for the fiscal year starting Oct. 1.

"I cannot accept that as is," said Michel, who added that House Republicans also want larger defense spending cuts and smaller tax increases than the Senate committee proposed after rejecting Reagan's original budget.

To bring down projected deficits and hopefully interest rates as well, the Senate committee recommended a significantly larger tax increase than Reagan did last February, a smaller defense buildup and sharp new cuts in domestic programs including Social Security.

Michel's move came as Senate Republican leaders, faced with a backbenchers' revolt in their own ranks, moved to finesse an embarrassing vote that would also have put the GOP-controlled Senate on record against the Budget Committee's Social Security "solvency" proposal.

After several hours of behind-the-scenes negotiations with Sen. John H. Chafee (R-R.I.), a Republican moderate who objected to any Social Security cuts in the budget resolution, the GOP leaders agreed to a compromise pledging that "any corrective action . . . shall not include any more than is absolutely necessary to preserve the financial integrity of the Social Security system."

They also agreed to language asserting that Congress would await recommendations of a special commission that is studying Social Security solvency problems before taking any legislative action to curtail benefits or raise new revenues for the system.

GOP leaders said these changes only carried out their original purpose, but they were enough to satisfy Chafee at least for the time being.

Last night the Senate voted 51 to 44, with only three Republican defections, to table a Democratic proposal that would have rejected any Social Security cuts in the budget resolution. But another fight on the issue is expected from Chafee as well as the Democrats when the budget resolution comes up next week on the Senate floor.

While the news was mostly bad for Reagan on the budget front, Democrats were faring little better with their own version of a budget for fiscal 1983, proposed on Monday by House Budget Committee Chairman James R. Jones (D-Okla.).

House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) gave it only a lukewarm blessing; it ran into objections from both Democratic liberals and conservatives, and even Jones appeared skeptical about its chances on the House floor.

"My concern is that we won't be able to pass anything" in the House, said Jones, although Democrats on the Budget Committee said Jones probably has the votes to get it out of the panel. Jones' committee is scheduled to begin marking up the budget resolution today.

In the midst of the turmoil, Reagan reached out once again to Rep. Phil Gramm (D-Tex.), who helped engineer conservative Democratic support for Reagan's budget last year and is developing a plan for this year that roughly parallels the Senate Budget Committee proposal.

Reagan called Gramm to congratulate him on his recent Democratic primary victory and in the process told Gramm he was "pleased" by what he had heard of Gramm's plan for this year, according to deputy White House press secretary Larry Speakes. At least some of the proposals in the plan would be "acceptable" to the administration, Speakes said.

But to the extent that Gramm's proposal parallels the Senate Budget Committee plan, including some cuts in Social Security, it appeared to be stirring little interest among House Republican leaders, who bowed to many of Gramm's ideas last year.

Asked if Gramm was speaking for the House Republicans, Michel responded, "No, not the last I heard." Michel himself was meeting with Republicans with hopes of coming up with a budget alternative, aides said.

Although the Social Security cuts amount to only a relatively small share of the roughly $400 billion in deficit reductions that Reagan and the Senate Budget Committee have proposed for the next three years, they continue to dominate congressional budget deliberations, threatening to undercut the solid Republican base upon which Reagan built his successful bipartisan budget coalition last year.

"As much as I would personally like to see us do something on Social Security, there's just no way you can address that issue before the commission's recommendations," Michel told reporters after a meeting with moderate Republican "Gypsy Moths," who object to any Social Security cuts in the budget resolution.

An aide to Michel said Michel and House Republican Whip Trent Lott (R-Miss.) gave the same message to White House aides in a meeting on Capitol Hill later in the day--no cuts in Social Security.

But, at the White House, Speakes minimized the significance of Michel's comments. Another White House official said the administration had gone for Social Security savings in the budget only because the Senate Budget Committee wanted them included.

As for Jones' proposal, O'Neill indicated it probably can't pass the House on the strength of Democratic votes alone.

Implicitly conceding the likelihood of Democratic defections, probably from both left and right, O'Neill said the Democrats would employ Reagan's 1981 budget strategy in reverse. "As they used conservative Democratic 'Boll Weevils,' we're going to use the 'Gypsy Moths' this year," said O'Neill.

But he also indicated his coolness toward the contents of Jones' proposal, which would cut domestic programs, although not as much as under the Senate plan. "I think we're giving quite a bit, to be perfectly truthfully," said O'Neill, reiterating that Jones' plan is the absolute "bottom line" for Democrats.