House Republican leaders, groping for a way to reduce deficits while appearing to shield Social Security from budget cuts, reverted to creative accounting yesterday and urged Congress simply to stop counting the financially strained Social Security system as part of the budget.

The novel proposal by the Republicans, bedeviled by the Social Security issue ever since President Reagan embraced a Senate Budget Committee plan that includes $40 billion in unspecified steps to assure the "solvency" of the system over the next three years, came as the White House also hedged a bit on that Senate budget plan and the House Budget Committee neared completion of a budget of its own that would leave Social Security untouched.

Before ending a 15-hour session at 1:20 a.m., the Democratic-controlled House committee brushed aside a proposal by Rep. Jack Kemp (R-N.Y.) to drop Social Security from the budget and, voting 16 to 12, largely along party lines, decided instead to ask the Ways and Means Committee to look at the issue.

"We ought to take all spending out of the budget," complained Rep. David R. Obey (D-Wis.), ridiculing Kemp's proposal. "We could then become the 'off-budget committee.' "

On the substance of the budget, the committee started by endorsing a cut of $46 billion from Reagan's huge military buildup over the next three years, more than double the cut proposed by the Republican-run Senate committee last week. For 1983, the cut in actual spending was about $9 billion, leaving $212.3 billion for defense. This was roughly the defense cut figure recommended by Committee Chairman James R. Jones (D-Okla.) in a budget draft that has been cautiously embraced by House Democratic leaders.

The committee generally followed Jones' outline, which differed from the Senate draft in calling for more tax increases and defense spending cuts and fewer domestic program reductions.

By a mainly party-line vote of 17 to 13, the committee approved Jones' proposal to increase taxes $147 billion over three years, spurning an effort by Kemp to reduce the size of the cut. Jones' tax increase is $52 billion more than the Senate committee proposed for the same period.

The combination of spending cuts and tax increases in Jones' proposal would produce a deficit of about $104 billion next year, declining to $35 billion by 1985.

But the panel also added $400 million for transportation, restored full cost-of-living increases for veterans' benefits and railroad retirees (while limiting increases in federal workers' pay and pensions to 4 percent, as Jones proposed) and allowed room for restoration of 13 weeks in extended unemployment benefits that were dropped last year. In addition, the space shuttle got $300 million from the defense budget.

The committee was expected to approve the budget later today, but the plan will probably face far greater obstacles in the full House from Democratic liberals and conservatives as well as Republicans, who were still trying to draft a budget of their own.

As outlined by House Minority Leader Robert H. Michel (R-Ill.), the GOP proposal would take the three major Social Security trust funds and their projected combined deficit of $10.1 billion out of the fiscal 1983 budget, thereby reducing the nominal budget deficit by that amount and taking Social Security off the budgetary chopping block for now.

It would also reduce the apparent size of the budget itself, by eliminating more than $200 billion in Social Security and Medicare outlays from the spending total, now expected to be more than $750 billion.

But it would solve nothing of substance. Social Security would still be short of money that would have to be made up by benefit cuts, Social Security tax increases or other means that are under study by a special commission that is under orders to report back to Congress later this year. And the government's so-called "off-budget" deficit--made up of all the programs Congress has chosen not to count as part of the budget over the years--would be $10.1 billion higher.

"It won't hurt Social Security; it won't help it," said Rep. J. J. (Jake) Pickle (D-Tex.), chairman of the House Ways and Means subcommittee on Social Security. "It's just a way to make the deficit look smaller and the Republican budget look better."

"At least it gets one volatile, sensitive issue laid aside," said Michel, whose proposal dovetails with one that was floated last week by Senate Republicans who are up for reelection this fall and are nervous about any preelection Social Security cuts.

Social Security was included in the so-called unified federal budget in 1968, at a time when its coffers were flush and it helped President Johnson reduce budget deficits while he was spending heavily for the Vietnam war. But its inclusion is also supported by people like Congressional Budget Office Director Alice M. Rivlin, who take the nonpolitical position that a federal budget should reflect all federal spending. It would be a "poor budgetary practice" to take Social Security out of the budget, Rivlin said yesterday in response to a reporter's inquiry.

The White House, which gets burned by the Social Security issue every time it touches it, took a noncommittal stand on Michel's proposal. It is "regarded as a political issue without any real effect on the budget and the budgetary problem," said Edwin L. Dale Jr., spokesman for the Office of Management and Budget.

Meanwhile, the Senate succeeded in wiggling out of its most immediate Social Security predicament, voting 91 to 7 in favor of a rider attached to the defense authorization bill that puts the Senate on record as wanting no more "corrective" actions on Social Security than are "absolutely necessary" to keep the system solvent.

This was a compromise engineered by Senate Republican leaders to keep skittish GOP moderates who are up for reelection this year from voting for a Democratic proposal to knock the $40 billion in Social Security "solvency" measures--which most senators nervously interpreted as likely benefit cuts--out of the pending Senate budget.

The vote followed a heated debate in which Democrats accused Republicans of "obfuscation" and trying to solve Social Security problems with "smoke and mirrors," while Republicans accused Democrats of pushing a "do-nothing" strategy to help them get reelected.

Further attempts to knock out the $40 billion savings proposal are expected, however, after the Senate takes up its version of the budget on Friday.

In the House Budget Committee, tempers also flared over a proposal by Kemp to lop about $30 billion off the $182 billion deficit that both the administration and congressional leaders assume will occur next year unless a combination of spending cuts and tax increases is approved.

While Kemp argued that the deficit projection is unrealistically high, exerting undue pressure on Congress for tax increases, Democrats contended that Kemp was simply trying to fudge the deficit to protect the Reagan tax cut, which Kemp helped inspire.

"If we play that kind of game with the American people," said Rep. Leon E. Panetta (D-Calif.), "we're going to lose whatever credibility we may have left."