With Democrats forcing its hand, the Republican Senate interrupted consideration of a spending cut bill yesterday and passed a major spending increase, an anti-recession provision to give extra aid to the thousands of workers who have exhausted their regular unemployment insurance.

Voting 84 to 13, the Senate approved a resolution calling for the extended benefits while considering a bill that would make $12.7 billion in deficit reductions, mainly spending cuts, over the next three years.

The measure then was passed, 73 to 23, and sent to the House, which is considering similar deficit-reducing steps in piecemeal fashion.

The Senate bill would save $5 billion over the three years by imposing a 4 percent-per-year cap on cost-of-living increases in pensions paid to federal retirees, a provision the House has already rejected. Other major Senate savings include $2.5 billion from food stamps and $1.5 billion from a freeze on dairy price supports.

But in other action yesterday, the Senate refused to hit milk producers any harder by rejecting, 65 to 33, a proposal to cut dairy price supports below their current level and then narrowly defeating 49 to 48 a "compromise" proposal to reduce price supports as production rises.

By voice vote, however, it bowed to wheat and other grain producers by approving a paid set-aside program under which farmers would be compensated for withdrawing land from production.

Budget officials said this would cost $240 million next fiscal year but would result in three-year savings of nearly $400 million by reducing production, which would raise prices and reduce the need for price support payments.

Yesterday's vote, together with $17 billion in cuts in Medicare, Medicaid and welfare that the Senate approved earlier, means the Senate has now agreed this year to $30 billion in spending reductions by fiscal 1985, or about $3 billion more than it was ordered to do in the budget resolution that Congress adopted earlier this summer.

These cuts are on top of the reductions Congress voted last year. The House is expected to come up with less than the Senate this year and the two houses will have to settle their differences in conference.

Already in conference are the Medicare, Medicaid and welfare cuts voted earlier, together with the three-year, $98.9 billion tax increase the Senate also approved.

In that conference yesterday, representatives of the two houses agreed on an increase of about $6.5 billion over the next three years in unemployment taxes paid by business, and to about $2.6 billion in various Medicare and Medicaid cuts, including some that would require poor people receiving Medicaid to pay for some services that are free.

On the Senate floor, Republicans held the line against all efforts to restore spending until yesterday when they acknowledged that Sen. Howard M. Metzenbaum (D-Ohio) was on the verge of victory with a multibillion-dollar proposal for extended unemployment compensation benefits.

When a nose-count was made late Wednesday night, "we came up a little short," Senate Finance Committee Chairman Robert J. Dole (R-Kan.) said. GOP sources said at least six Republicans would have jumped ship on the issue.

After a flurry of meetings on and off the floor, Dole and other GOP leaders reached agreement with Metzenbaum on a compromise, which took the form of a "sense of the Senate" resolution calling on House-Senate conferees on the tax bill to push for a two-pronged liberalization of the unemployment program. It would:

Postpone new rules that Congress approved as part of its budget cuts last year and that are scheduled to take effect in September. This postponement would let the government continue paying extended unemployment benefits--13 weeks on top of the basic 26 to which insured workers are entitled--in states that are now eligible for the extended program or were on June 1. Without such a provision, a dozen states and hundreds of thousands of workers would be dropped from the extra 13-week program in September.

Add at least 10 and possibly 13 additional weeks of eligibility to the 39 now available in the hardest-hit states, for a possible total of 52 weeks in all.

Details of the plan had not been worked out as of late yesterday, but Republican leadership aides said its cost was expected to be between $1.3 billion and $1.9 billion a year, depending on what unemployment level is used to determine whether states are eligible.

They said the cost should not increase deficits because it would be offset by a proposed increase in income taxes for higher-income recipients of unemployment benefits.

Republican support for the Metzenbaum proposal was made easier by the fact that the Reagan administration indicated Wednesday for the first time that it might reverse itself and support some extension of unemployment benefits. The administration had said only a week before that it opposed any extension, but, as more and more workers face cut-off of their jobless benefits as the Nov. 2 elections draw near, political pressure for congressional action has grown intense.

It was not a Republican who led the fight against the Metzenbaum proposal, but Sen. Russell B. Long (La.), ranking Democrat on the Finance Committee.

Warning that a Senate proposal to add spending to the tax and spending cut proposals in House-Senate conference could lead to a deluge of House add-backs, Long said the likely result was a "Christmas tree to end all Christmas trees."

Long readily acknowledged his familiarity with well-decorated tax bills. "I know something about a Christmas tree because I was the author of the original Christmas tree," he said. But, he added, "Enough is enough . . . . We ought to stop this mischief."

Moreover, Long argued to no avail, a proposal to increase spending does not belong in a bill that is supposed to lead to less spending and reduced federal deficits.

Metzenbaum contended that the extended unemployment benefits were urgently needed, arguing that yesterday's action was proper because those thrown out of work by the recession, which he attributed in large part to current economic policies, shouldn't be required to bear the burden of reducing government deficits.

"These are people who played by the rules . . . and now that they need the help of the federal government it is not forthcoming," Metzenbaum said.

Approval of the Metzenbaum-Dole compromise not only saved the Senate Republican majority from a potentially embarrassing defeat but also may help Dole and other backers of the trouble-plagued tax increase bill get it out of conference and through the Democratic-controlled House, sources said. Some members who might have voted against a tax increase may find it more difficult when a benefit increase is included.

The politics of the situation were not lost on Sen. Ernest F. Hollings (D-S.C.), who congratuled Dole on "getting his party and his program saved in one fell swoop."

All Washington area senators except Harry F. Byrd Jr. (Ind.-Va.) voted for the unemployment benefits extension.

As the Senate was wrapping up its spending cuts package, the House took a step closer to budget compliance by accepting a Republican proposal to save $2.1 billion over three years by requiring lump-sum, payment in advance of insurance premiums for Federal Home Administration loans, which are currently stretched out over the life of the loans.