The House, lining up behind its Agriculture Committee's new five-year farm bill, yesterday rejected an amendment pushed by a grass-roots farmer movement that would have established mandatory farm production controls to boost prices and reduce crop surpluses.

The controversial mandatory program, promoted by the American Agriculture Movement and sponsored by Rep. Harold L. Volkmer (D-Mo.), was defeated on a voice vote that allowed members to avoid formally going on the record. The same proposal had failed earlier in the committee.

Volkmer argued that his amendment would substantially raise prices that farmers received for wheat and feed grains, while adding only pennies to the retail cost of consumer food items. Opponents, including a number of committee members, contended that domestic agribusinesses and export sales would suffer from reduced farm output.

Much of yesterday's floor debate was a replay of the arguments discussed at length during weeks of committee deliberations on the legislation. Throughout the day committee members lined up against one another on a series of votes.

In the closest vote, the House defeated an effort by committee renegades to set up a "marketing loan" program that would have allowed farmers to repay price support loans on their crops for less than the value of the loan. The vote was 228 to 200.

That approach, pushed by Rep. Arlan Stangeland (R-Minn.), also included a provision that would have targeted federal income support assistance to farmers in the greatest economic need. Stangeland and his allies argued that the amendment would reduce federal surplus costs by moving grain onto the market at whatever price it would bring and, at the same time, raise farm income.

The amendment provoked extended debate, highlighted by a plea from Agriculture Chairman E (Kika) de la Garza (D-Tex.) that "this is not the time to gamble, as enticing as this legislation sounds."

On the final vote, 79 Republicans joined 121 Democrats in support of Stangeland, while 102 Republicans and 126 Democrats voted against.

The House also rejected two other efforts to amend the committee bill.

One, by Rep. Barney Frank (D-Mass.), would have cut farm income supports set by "target prices." Frank, noting that the Reagan administration supported him, argued that the committee bill would provide more help to the biggest farmers.

But farm-state opponents insisted that the committee bill's proposed freeze on income supports was barely adequate to protect farmers suffering economic distress. Frank was defeated, 334 to 93.

A more mischievous amendment by Rep. Robert F. Smith (R-Ore.), intended to upset a farmer referendum provision contained in the bill, would have given farmers a chance to vote approval or rejection of whatever farm bill Congress produces.

"If it is so desirable to let them vote, what's wrong with letting them vote on a farm bill this year?" Smith asked. "If farmers agree, then that's the farm bill; if not, Congress in 1987 would have to go back to work on a new farm bill."

Smith was defeated, 318 to 108, which left the House with one major pending item of business in its wheat and feed grain section -- an administration-supported amendment by Rep. Edward R. Madigan (R-Ill.) to kill the committee provision that would allow farmers to vote on a production control program.

A vote on the Madigan amendment is expected today, with the House Democratic leadership promising that work on the measure will be done by week's end.