Twelve farm-state Republicans teamed up with Senate Democrats yesterday to hand the Reagan administration a key setback in its effort to cut the costs of a heavily over-budget farm bill.

But the surprisingly close 51-to-48 vote against a proposal to freeze for one year the direct income-subsidy payments to grain, cotton and rice farmers presaged more intense debate and lobbying over the controversial bill. The move to freeze the so-called deficiency payments at current levels for one year and then reduce them by 5 percent in succeeding years was sponsored by Sen. Richard G. Lugar (R-Ind.), who said it would save more than $7 billion over the three years.

As reported by the Agriculture Committee, the bill provides for a four-year freeze on the payments -- a step that farm-state senators insisted was essential for maintaining farm income during a time of severe economic stress in agriculture.

But the administration, Majority Leader Robert J. Dole (R-Kan.) and Agriculture Committee Chairman Jesse Helms (R-N.C.) warned that unless the four-year freeze was removed, the bill faced a veto.

Republican senators from the economically troubled Midwestern grain regions, however, argued that the income supports were crucial to keeping family farms going and to sustaining them until depressed prices improve.

"The Lugar amendment makes a political decision, not an economic decision," Sen. Charles E. Grassley (R-Iowa) said. "In my state, 20 to 30 percent of the people who are now in farming are no longer going to be farmers -- that amendment is a political decision that would remove them from agriculture."

Grassley said a four-year freeze on income supports would provide "certainty and predictability" that would allow agricultural lenders to sustain their farm borrowers.

However, Helms, who opposed the bill when it was reported from his committee, contended that the targets used to set the deficiency payments were "an arbitrary standard used to determine how much subsidy is to be transferred to the farm sector . . . . These fall into the category of uncontrollable costs."

Latest administration calculations say the cost of the pending farm bill will be at least $60 billion -- $5 billion more than when it was reported from the committee last month. The administration attributed the new, higher figure to declining exports and continued low prices, raising the possibility of more government acquisition of surpluses. The congressional budget resolution earmarked $35 billion for the farm support programs.

Sen. John Melcher (D-Mont.), author of the committee provision for the four-year freeze, agreed that savings had to be found in the farm bill, but said the deficiency payments were not the place to look. He promised to offer an amendment that would require the Senate to cut the bill by at least $7 billion. Melcher said the deficiency payments, calculated on the basis of the difference between market prices and farmers' production costs, are vital "to keep life in American agriculture."

Another Democrat, J. James Exon of Nebraska, urged the Senate to tread cautiously on cutting the payments. "Understand this amendment for what it is," he said. "It is the administration's program -- lock, stock and barrel. If this becomes law, we'll rue the day it ever passed."

The final vote on the Lugar amendment found eight Democrats, including six from the urban Northeast, joining the administration's budget-cutting campaign. Ten of the 12 GOP defectors were from the Midwest.

Dole, who had predicted no more than 25 votes for the Lugar amendment, said he was "somewhat encouraged that it was as close as it was . . . . Maybe it is time to sit down on both sides of the aisle and work out a farm bill."

But that prospect seemed distant, with a raft of other controversial administration amendments aimed at cutting farm-income subsidies waiting offstage.

An effort will be made to limit the freeze to two years; other moves are planned to try to reduce the amount of individual subsidies.

The Senate yesterday continued in stalemate over a controversial amendment dealing with cargo preference -- the law that requires at least 50 percent of government-generated shipments, such as the food aid programs, to move on U.S. flag vessels.

Agricultural interests contend that the cargo-preference law hamstrings efforts to expand farm exports, which otherwise could be shipped at the less expensive rates offered by foreign merchantmen. The situation became more complicated after a federal judge ruled earlier this year that all government grain shipments must move under cargo preference.

A compromise that would break the logjam caused by the cancellation has been blocked by senators from Great Lakes states who insist that the plan would doom many of their shallow-water ports. The compromise, sponsored by Sen. Thad Cochran (R-Miss.), would raise the cargo-preference requirement to 75 percent for food-aid shipments but would exempt other government-sponsored grain-export programs.

The impasse was broken last night with Senate acceptance of an allocation proposal aimed at maintaining Great Lakes' grain shipments at 1984 levels if possible.