After months of inconclusive efforts by the White House and Congress to find a magic formula for deficit reduction, the Senate Budget Committee arrived at what many lawmakers considered the key question:

Would the committee risk offending 36 million retirees by eliminating next year's cost-of-living increase for Social Security benefits to demonstrate its determination to cut spending? Or would it flinch and vote to keep the increase?

The hearing room fell silent as the roll was called. When the voting was over, the committee had voted no -- on both questions.

In voting against proposals both to deny the increase and to grant it, along with another one to pay the increase to just the most needy of the nation's elderly, the committee made an eloquent, if somewhat embarrassing, statement about where Congress is on the deficit issue.

Lawmakers fear and deplore the huge budget deficits of $200 billion a year or more for the rest of the decade. They worry that the deficits will eventually abort the economic recovery and possibly their political careers.

But Congress can't seem to come to grips with specific spending cuts -- to say nothing of tax increases -- that will be needed to bring the deficits under control.

And there is increasing doubt among many lawmakers that they can do so without active participation by President Reagan to lead the way to a compromise, especially on defense and Social Security, maybe even on taxes.

Reagan has said he will consider a Social Security proposal if it has overwhelming bipartisan support, but lawmakers say that is not enough of a push. He is resisting any more than the modest defense restraint already included in his budget and remains adamant against tax increases.

The difficult task of writing a massive deficit-reduction package in this climate was undertaken by the Republican-controlled Senate Budget Committee almost by default. The White House set a target of deficits no higher than $100 billion by fiscal 1988 but failed to come up with the spending cuts to meet it. Senate Republican leaders also tried to meet the target and failed. Democratic leaders haven't even tried.

In theory, the Budget Committee only sets targets for taxes and spending. But, in fact, it is writing the first draft of a deficit-reduction package by setting targets that later would be translated into spending-cut legislation, possibly with modifications negotiated between congressional leaders and the White House.

The budget panel did vote to take an $11 billion bite out of the president's $284 billion defense budget for next year. But the move came relatively easily in light of lawmakers' growing sense of public support for reining in the Pentagon after four years of massive increases in defense spending, punctuated by reports of waste, malfunctioning weapons and excessive charges by defense contractors.

The real test came on the domestic spending side, and it was here that the committee ran for cover.

With domestic spending programs having borne the brunt of earlier Reagan budget cuts and with many of Reagan's new proposals aimed at middle-income people, who tend to vote in large numbers, cutting the domestic side of the budget has become increasingly difficult.

One after another, spending cuts and program eliminations proposed by the president in his fiscal 1986 budget were rejected, often by substantial bipartisan majorities, with Democrats taking the lead and Republicans dutifully providing enough votes for them to prevail.

Amtrak, the Job Corps and Legal Services Corp. were spared a heavy chop from the budget ax. Medicare recipients, children on the school lunch program and families who want to vacation in federal recreation areas were saved from digging into their pockets to pay new charges for services. Programs for the poor, which were cut with hardly a pause four years ago, got special treatment this year, protected even from the spending freeze that the committee was applying to most other programs.

This does not mean that the committee, with its 12-to-10 Republican majority, including some of the Senate's most conservative members, had gone on a spending spree.

What it was doing was voting in nearly every case for a spending freeze, sometimes at current levels, other times at current levels adjusted for inflation. It drew the line against nearly every proposal that would reduce spending below a freeze level, to say nothing of proposals to kill whole programs.

As of late Thursday, having gotten most of the way through the budget, the committee had made deficit reductions of $21.1 billion from defense and $11.3 billion from domestic programs, mostly by freezing broad categories of spending.

But this was nearly $20 billion short of Senate Republican leaders' goal of at least $50 billion in deficit reductions next year as the first step toward reaching the $100 billion per year deficit goal over the next three years.

As measured against a plan by committee Chairman Pete V. Domenici (R-N.M.) to meet the goal, the committee exceeded the defense target by $2.5 billion but fell $22 billion short on the domestic side.

Thus, even if the committee's tentative votes of last week are not shattered by its Social Security impasse, it would be only two-thirds of the way home in spending cuts.

In the past, a freeze and nothing more would have been an extraordinary achievement.

Before Reagan, a freeze would have been unthinkable except to rock-hard conservatives. Last year, a freeze proposal was noteworthy when it got the support of one-third of the Senate. But now, as Budget Committee member Robert W. Kasten Jr. (R-Wis.) noted last week, it has become a virtual "cop-out" position as senators embrace it to avoid deeper spending cuts.

The problem is that the ground has shifted under the committee. Deficits have grown so big in proportion to discretionary spending that a freeze simply doesn't help much any more.

Even the roughly $32 billion proceeds from a freeze, once enough to balance a budget, are now barely enough to keep the deficit under $200 billion. To get the deficit on enough of a downward trend to reach the $100 billion target by 1988, it would take a freeze, including Social Security, plus almost all of Reagan's drastic program cuts.

This "freeze-plus" approach is what Domenici proposed; he got the freeze but was defeated on almost every vote on a "plus." So many Republicans, especially those who are up for reelection in two years, joined the Democratic minority in resisting spending cuts that the Democrats were in effective control of the committee most of the time.

The Democrats and dissident Republicans remain critical to resolving the Social Security issue and moving on to completion of the deficit-reduction plan. So long as they block passage of the freeze and insist on forcing a sure-loser vote on preserving the status quo, they have leverage to bargain for other changes in the budget, such as more defense cuts or tax increases.

Social Security is important leverage because loss of the freeze would knock $6 billion off Domenici's list of potential deficit reductions for next year, the largest single savings on the domestic side of the budget. Even more important, it would be difficult to freeze other inflation-indexed programs, such as government workers' pensions, if Social Security is exempt.

And the whole freeze concept could fall apart if Social Security, as a critical linchpin, is removed. One of the main arguments for a freeze is that it is almost the only logical framework for getting at inflation adjustments for benefit programs.

But the Democrats also run into political risk in moving too far out front in a departure from the party's general strategy of letting Republicans take the lead -- and the heat -- on spending cuts and, if necessary, tax increases.

The White House moved quickly on Friday to chide the committee for falling short of spending cuts recommended by Reagan, and can be expected to have a lot more to say if the Democrats prevail in using the Social Security stalemate to force consideration of tax increases.

Politics aside, what was interesting as the committee sputtered toward a weekend recess Thursday night was the increasing discussion of tax increases, from Republicans as well as Democrats.

Both Domenici and the ranking Democrat on the Budget Committee, Sen. Lawton Chiles (D-Fla.), said yesterday that the panel is "moving very quickly toward more taxes" if the lawmakers keep balking at major spending cuts.

Domenci, appearing on CBS-TV's "Face the Nation," said he would oppose any such new taxes. He cautioned that it is early in the game and predicted that the lawmakers will get a "dose of reality" after this first round of sparring.

He said that approving any new corporate taxes now might jeopardize later action on tax reform.

Chiles said on the same show that, "If you want to get to these [deficit-reduction] goals, there's going to have to be taxes."

Tax increases are hardly mentioned in Congress these days in light of Reagan's adamant opposition to them and his ability, demonstrated in last fall's campaign, to annihilate those who propose them.

But, with Budget Committee members under the gun to fill the gap between spending cuts and the deficit reduction target with something, taxes -- such as corporate levies suggested in connection with tax simplification legislation -- are an inviting target.

"The basic issue we're setting up is spending versus taxes," said committee member John C. Danforth (R-Mo.) on Thursday night as it became clear the committee would not meet its target for domestic cuts. Some other members, including one who is adamantly opposed to tax increases, said there is at least some chance that tax increases will be seriously considered before the committee completes its work, theoretically by week's end.