The Senate Budget Committee deadlocked over Social Security last night as it rejected a proposal to freeze benefits next year but left open the possibility of reopening the issue and linking a benefit freeze with a tax increase or further defense-spending cutback.

The stalemate over Social Security came after the Republican-controlled committee voted along bipartisan lines to reject President Reagan's proposals to end federal subsidies for Amtrak, sharply restrict student loans and wipe out or drastically reduce a variety of other programs ranging from the Job Corps to Urban Development Action Grants.

The committee also rejected the president's proposal to raise bills for Medicare beneficiaries, dropped his proposed work requirements for food stamps, rejected his proposed cutbacks in child nutrition programs and voted not to disband the Legal Services Corp., as Reagan had proposed.

Technically, the committee was working on a budget resolution for next year that would set targets for taxes and spending. But, with the failure of both the White House and Senate Republicans to come up with a deficit-reduction package that would halve annual deficits of $200 billion or more over three years, the committee is attempting to do the job itself.

It was making some progress, although falling short of targets for domestic-spending cuts, until it hit the controversial issue of Social Security. Then it went in all directions at once.

Voting 12 to 9, it rejected a proposal from committee Chairman Pete V. Domenici (R-N.M.) to eliminate cost-of-living increases for retirement benefits for one year, along with a proposal from Sen. Ernest F. Hollings (D-S.C.) to give the inflation adjustment next year only to low-income retirees.

But the panel also voted twice, first 13 to 8 and then 12 to 9, against leaving Social Security inflation adjustments alone, thereby indicating it wanted some change.

Several members, Republicans as well as Democrats, indicated they might vote for a freeze or some other cost-of-living modification in exchange for larger defense spending cuts than the committee approved earlier in the week or tax increases, which the committee had not been planning to consider.

The Social Security issue is expected to be reconsidered later when the committee will also consider freezing inflation adjustments for federal retirees next year. Domenici refused to bring up the government retirees' cost-of-living increases last night, saying, "I don't think we should freeze COLAs [cost of living adjustments for other retirement programs] if we're not going to do it for Social Security."

Neither Reagan nor Democratic leaders have encouraged the idea of cutting Social Security as part of the deficit-reduction effort, and members of Congress are skittish about tampering with the huge, popular federal retirement program. In yesterday's vote, seven Republicans favored the freeze while four opposed it. Eight Democrats voted against it.

After two days of work on the domestic side of the budget but before the committee took up Social Security, it had approved $9.1 billion in domestic-spending cuts for programs considered so far -- $16.6 billion short of Domenici's target of $50 billion to $60 billion in deficit reductions this year.

The committee had begun its deliberations by voting a $20 billion reduction in defense spending Tuesday, allowing the Pentagon a spending increase sufficient only to keep pace with inflation, estimated at about 4 percent next year. But lawmakers bogged down in trying to cut popular domestic programs.

Except for about $2.5 billion in cuts from rural housing, small business loans and postal subsidies, the committee generally stuck to freezing spending at current levels or, in some cases, at levels that would cover the cost of inflation next year.

In addition to voting to protect the Amtrak rail passenger system and guaranteed student loans, the committee spurned Reagan's proposals to kill the Job Corps, impact aid for school systems serving areas with large numbers of federal workers, the Economic Development Administration, the Appalachian Regional Commission, urban development action grants, mass transit and air carrier subsidies, community services block grants, small business loan guarantees and community development loan guarantees.

Yesterday's voting continued a pattern established earlier in the week when Democrats took the lead in pushing for rejection of Reagan's spending cuts and won enough Republican support to prevail.

On guaranteed student loans, the committee rejected Reagan's proposal to cap payments at $4,000 a year, restrict eligibility for loans to families with annual incomes of less than $32,500, require recipients to contribute at least $800 a year to their own education and increase students' interest costs.

Medicare savings were limited largely to a freeze on payments to hospitals and doctors. Reagan's proposals to increase out-of-pocket costs to Medicare patients were rejected by a rare unanimous vote in which even Domenici, who has tried to hold the line for Reagan's cuts, voted to reject increased costs for beneficiaries.

Similarly, Reagan's proposals to shift Medicaid costs to states were shelved; Medicaid costs were allowed to grow to keep pace with inflation. Other health programs were frozen at current levels for next year and allowed to grow by 3 percent a year thereafter.

For education and jobs, the committee rejected all of Reagan's program cutbacks and eliminations and instead voted to allow all programs to grow with inflation.

Just as it cut but did not eliminate the Small Business Administration loan program Wednesday night, the committee rejected Reagan's proposals to terminate SBA disaster assistance. It also rejected cutbacks in other community aid programs, voting instead to freeze them at current levels for a year.

On transportation, the committee rejected Reagan's proposals to kill mass transit and Amtrak subsidies, voting instead to freeze Amtrak funding at $700 million a year and restrict transit operating and capital subsidies to current spending levels.