The Senate yesterday voted heavey cuts in fiscal 1980 domestic spending, but then undid them with increased outlays for defense, and ended up with a likely $31.6 billion federal deficit.

The Senate Budget Committee had warned that increasing the budget would fuel inflation. It had recommended a deficit of $28.4 billion with only a modest increase in defense spending to take inflation into account.

The day's trade-off started with an overwhelming vote, 90 to 6, to order $3.6 billion in domestic spending cuts to help make up for cost-saving legislation that Congress promised earlier but did not enact.

But, following President Carter's lead, the senators then voted 78 to 19 to increase defense outlays next year by $3.2 billion more than the Budget Committee had recommended, largely to placate conservatives who have made higher defense outlays the price of their votes for the strategic arms limitation treaty with the Soviet Union.

The defense increase pushed the projected deficit for the year beginning Oct. 1 over the politically sensitive $30 billion mark that Senate leaders have hoped to avoid for fear of jeopardizing the budget's approval.

The Senate now will consider whether to make room in the budget for an anti-recession tax cut. However, the spectre of the larger deficit was expected to discourage some lawmakers from supporting a tax cut.

The action came as, separately, the House prepared for floor votes today on its own version of the budget reso- lution, with similar battles expected on defense and taxes, and a separate challenge from a Republican "alternative" budget.

The House GOP Policy Committee gave formal approval yesterday to a proposal that would hold the deficit to $20.2 billion-- compared to $29.2 billion in the pending House resolution-- with a $20 billion tax cut to boot.

The GPO is proposing $400 million more than the House Budget Committee in defense spending and $20 billion in reduced spending levels for a variety of categories, including transportation and foreign aid.

The minority party also uants sizable cuts in unemployment benefits and welfare payments. GOP sources said these stemmed from assumptions that the economy would pick up as a result of the tax cut-- not from any cutbacks in domestic programs.

The vote in the Senate to slash domestic spending was a victory for that chamber's Budget Committee, which has recommended the reductions to preserve the integrity of the congressional budgetmaking process.

The lawmakers had approved a $23 billion deficit target last spring, but based that on $5.6 billion in promised economies, such as Carter's hospital cost-containment bill, which later were not passed.

Budget Committee Chairman Edmund S. Muskie (D-Maine) had argued that unless the Senate cut its fiscal 1980 appropriations bills to make up for that, it would push up the deficit sharply, breaking Congress' pledge of fiscal austerity.

The series of votes is part of the five-year-old congressional budget process. Congress sets initial spending targets in the spring, then revises them in September to form binding ceilings on outlays.

The vote to cut $3.6 billion from the appropriations bills was supported by two key committee chairmen who earlier had threatened to block this "reconciliation" procedure-- Sens. Russell B. Long (D-La.) and Warren G. Magnuson (D-Wash.).

Long, chairman of the Finance Committee, and Magnuson, head of the Appropriations Committee, had said it would not be feasible to make the cuts. However, both agreed to a leadership compromise.

The vote on defense spending uould boost Pentagon outlays for fiscal 1980 a full 3 percent above fiscal 1979 levels, after adjustment for inflation-- in line with what Carter promised European leaders two years ago.

At the same time, however, the Senate voted to boost defense spending for fiscal 1981 and 1982 by 5 percent after inflation, ignoring objections from both the Budget Committee and the White House, which say 5 percent is too much.

In a floor speech opposing the defense hike, Muskie argued it would be unfair to raise military outlays while forcing domestic programs to bear the brunt of the budget austerity program.

The Senate is expected to complete action on the budget resolution today, after voting on the tax-cut question. After the House finishes its work, the two versions will have to be resolved by a conference.