The Senate Finance Committee threatened yesterday to scuttle President Reagan's proposal for a free-trade agreement with Canada, in an apparent sign of growing bipartisan congressional frustration over White House refusal to support broad-based trade legislation.

Committee members did not vote on the issue, but voiced strong opposition to giving the administration authority to negotiate a free-trade pact with Canada. The committee's opposition surprised U.S. Trade Representative Clayton Yeutter, the Canadian Embassy and even members of the committee.

"We all thought it was our own personal view, but found out that everyone felt the same way," said Sen. Lloyd Bentsen (D-Tex.), the No. 2 Democrat on the committee.

Committee staff members said 14 of the 20 panel members opposed the free-trade pact with Canada, an administration initiative that arose from Reagan's 1985 "Shamrock summit" with Prime Minister Brian Mulroney. Yeutter called it "the single most important trade agreement that has been taken in this country probably in this century."

Only Sens. Daniel Patrick Moynahan (D-N.Y.) and John Chafee (R-R.I.) firmly supported the administration position.

Reasons for the opposition varied. At least six committee members -- including Chairman Bob Packwood (R-Ore.) and Max Baucus (D-Mont.) -- voiced concern that the administration would not protect American lumbermen, pig farmers, fishermen and potato growers, who are being hurt by soaring Canadian imports.

"We get the feeling we are being set up as a patsy," Packwood said. "We are not going to roll over on this. The committee has reached the end of its patience."

Baucus said: "We are worried that it will not be a true free-trade agreement. We are worried that we will end up with something that will help the Canadians more than it will help the United States."

Sen. John C. Heinz (R-Pa.) voiced similar concerns. "It will be difficult, if not impossible, for members of this committee to vote for a trade liberalization with Canada if we don't stick up for the United States more," he said.

Packwood scheduled a vote for next week, but committee sources said he might delay it another week.

Under special provisions of U.S. trade law, either of the two main congressial committees dealing with trade -- Senate Finance and House Ways and Means -- has until April 23 to deny the president negotiating authority to go ahead with the free-trade agreement.

Sen. George Mitchell (D-Maine) said he got angry when Agriculture Secretary Richard Lyng refused to give government aid to ailing potato farmers because it would offend the Canadians, only to have Canada help its farmers with subsidies.

Another group, including Sen. John C. Danforth (R-Mo.), chairman of the trade subcommittee, and Bentsen expressed dissatisfaction with administration trade policies, including a White House refusal to support broad-based trade legislation this year.

"There is a resentment on the committee on the way we have been treated on the trade issue," Bentsen said. "The administration has stiffed us all year long."

Danforth said his "strongest reason" for opposing negotiating authority for the Canadian free-trade pact is to force Reagan to deal with Congress on trade. "We are not in business right now," Danforth said. "The president is stonewalling us."

Yeutter was described by aides as "shocked and surprised" by the committee's position, but administration trade strategists began to try to turn members around.

Senate Foreign Relations Committee Chairman Richard Lugar (R-Ind.) sent a letter to Packwood last night reminding him of the foreign policy implications of turning down the free-trade pact.

Roger Bolton, a spokesman for Yeutter, said, "The administration believes the negotiations with Canada offer a historic opportunity. Our hope is that Congress will go along."

Canadian Finance Minister Michael Wilson, here for IMF-World Bank meetings, told an embassy press conference that he was "surprised and very concerned" at the "strength and vigor" of the opposition to the free-trade pact.

In Ottawa, Mulroney said Canada is not prepared to make specific concessions on ongoing trade issues before the negotations start.

The free-trade agreement has been front-page news in Canada for more than a year, and some U.S. officials have expressed concerns that the White House has not given it enough attention.

The United States and Canada have the largest two-way trade relationship in the world, totaling about $120 billion.