The Senate yesterday defeated on a 49-to-49 tie a move to cut congressional salaries by 10 percent as it ended a week of action on deficit reductions by modifying proposed cutbacks in Medicare and Medicaid.

After voting earlier to cut defense spending but to protect Social Security recipients from reductions in their cost-of-living adjustments, senators found themselves on the spot when Sen. Jesse Helms (R-N.C.) proposed a $7,510 cut in the pay of Senate and House members.

"This amendment proposes that Congress share the burden," Helms said as senators grinned and groaned at the unpalatable choice the maverick conservative had thrust upon them as they were preparing to quit early for Kentucky Derby weekend.

Many considered the move "frivolous," in the word of Sen. John D. (Jay) Rockefeller IV (D-W.Va.), who, as probably the Senate's wealthiest member, set off laughter among some of his colleagues as he cast his vote against cutting the $75,100 a year legislators are paid.

The savings of less than $4 million from the pay cut spread over 535 lawmakers would hardly nick the budget deficits of more than $200 billion that are projected for years to come. As such, the pay cut proposal was seen as a grandstand gesture by many members.

Always conscious of constituents' suspicions that they are overpaid and greedy for more, lawmakers are reluctant to deal openly with the subject.

This time Helms put the issue out in the open, and visitors in the galleries erupted with particular delight as Sen. Edward M. Kennedy (D-Mass.) and some other millionaire members appeared in the chamber to vote against the measure.

Some senators joined in enjoyment of the spectacle. As Rockefeller voted against the cut, Sen. Alan J. Dixon (D-Ill.) leaned back and howled with laughter until he was red in the face. "I love it," he said as he reached out to shake hands with Rockefeller, who looked as though he wished he were someplace else.

Another visibly uncomfortable member was Sen. John Heinz (R-Pa.), millionaire heir to a pickle-and-catsup fortune, who held out his vote to the end and then joined most other Republican leaders in supporting the cut. Helms' proposal was not debated and wound up losing on the tie vote. Although Vice President Bush had been on hand in case his vote was needed to break a tie earlier in the week, he was nowhere to be seen in the Senate yesterday.

Helms vowed to try to bring up the issue again next week. "Maybe people around the country will get in touch with their senators" over the weekend, he said after the vote.

In the vote on Medicare and Medicaid, the Senate agreed, 93 to 6, to restore $2.6 billion of the $20.1 billion that would have been cut from the two big health-support programs over the next three fiscal years under a deficit-reduction plan negotiated by the White House and Senate GOP leaders.

This left a cut of $17.5 billion, however, and Democrats, contending that the reduction is excessive, plan to try next week to restore more of the funds, including whatever is necessary to keep patients from having to pay more in health-care premiums.

Yesterday's move by Republican leaders appeared aimed at derailing the Democratic effort by giving wavering lawmakers a chance to vote for some fund restorations without significantly shrinking the anticipated savings for deficit reduction.

The new compromise approved yesterday would result in some increase in out-of-pocket costs for Medicare, the government's health program for the elderly, although not as much as the original GOP plan.

Premiums for care by physicians would rise from 25 to 30 percent, rather than 35 percent as the White House and GOP compromise had proposed, over the next five years.

Yesterday's vote retains the earlier proposal to freeze hospital and physician charges at current levels.

Under the new Medicaid agreement, a proposed cap on the federal share of health care for the poor would be eliminated, easing the pressure on states to assume a greater share of the overall cost or cut service.

In all, Medicare would be cut by $16.3 billion, rather than $18.1 billion, over three years. The three-year savings from Medicaid would be $1.2 billion, rather than $2 billion.

The modification was negotiated by Republican Sens. David F. Durenberger (Minn.), John H. Chafee (R.I.) and Heinz with Majority Leader Robert J. Dole (Kan.) and Finance Committee Chairman Bob Packwood (Ore.), who introduced it yesterday after Democrats, in some disarray over strategy, passed up a chance to move first.

"My hunch is it will stick," said Packwood, predicting it will hold up against the Democratic move for further modifications.

Packwood also contended that the Senate, despite flinching on some of the proposed cuts, will be able to match the $52 billion in deficit reductions for next year that were proposed in the original Reagan-GOP package. Dole, however, appeared less certain, suggesting it may fall short.

In other action yesterday, the Senate voted 71 to 27 against a proposal by Sen. Phil Gramm (R-Tex.) to kill the Legal Services Corp. and transfer its $300 million budget to farm programs.

Meanwhile, as expected, Reagan expressed strong opposition to the defense-spending cutbacks approved Thursday by the Senate, and White House aides predicted reversal of the 51-to-48 vote by which the Senate agreed to limit defense-spending growth to the rate of inflation next year. Reagan and Senate leaders previously had agreed to defense growth of 3 percent after inflation.

"It's an irresponsible act. We've already made the reductions we could make in military spending without reducing our ability to maintain the security we must have," Reagan told reporters in Bonn, West Germany, where he is attending a seven-nation economic summit.

"Our position is that this is a close vote and we may be able to turn it around," presidential spokesman Larry Speakes said. "The president feels strongly he's compromised enough."

Speakes said anything less than a 3 percent after-inflation increase could jeopardize defense contracts.

Although the Senate was still far from finishing with the deficit-reduction package after more than a week, GOP leaders were already working on fallback positions, including a complete substitute if the plan is totally unraveled. Compromises were being made on programs ranging from postal subsidies to small business aid, although farm program cutbacks reportedly still were a stumbling block.

After a meeting yesterday with Office of Management and Budget Director David A. Stockman, whose wife Jennifer had just given birth to their first child, Rachel Lauren, Stockman was quoted by Dole as saying, "I delivered. Why haven't you?"