Special Trade Representative Robert S. Strauss warned Congress yesterday that failure to extend the government's authority to waive penalty duties on subsidized imports would doom the latest round of trade-liberalizing talks now nearing completion in Geneva.

"This issue has become the key procedurally to our ability to cnclude an acceptable agreement," Strauss told the House Ways and Means trade subcommittee as it began consideration of the proposed countervailing duty waiver extension.

Strauss' claim that improved trade rules hinge on extension of the waiver was greeted skeptically by the AFLCIO, the only major group to oppose the extension, and by Rep. William J. Hughes (D-N. J.), whose district includes a butter cookie maker threatened by subsidized cookie imports from Denmark.

"Negotiations will continue," said Rudy Oswald, research director for the labor federation. "And if they don't, the U.S. Congress will be on notice that unfair subsidies are to be considered acceptable in trade rules."

To continue waiving duties will drive some companies out of business and serve as "just another sign to the rest of the world that this nation is a cream puff," said Hughes.

Although the amount of the waived duties is $47 million, relatively small in trade terms, European and other countries are threatening to hold off final approval of new trade agreements until they are assured the waiver will be extended.

The administration, apparently believing the threats, views the extension as critical to the four-year effort aimed at lowering both tariff and non-tariff barriers to trade, including the subsidies that the countervailing duties are designed to offset.

"Extending the waiver will merely afford the Congress an opportunity to examine the negotiating results and make up its mind," Strauss told the subcommittee. "Without the procedural extension we are discussing today, the Congress will not have that opportunity."

The controversy arises because the 1974 Trade Act, in laying the groundwork for the current round of trade talks, empowered the Treasury Department to make selective waivers of countervailing duties on an estimated $600 million in subsidized imports, but only through Jan. 3 of this year.

A bill to extend that authority died in the final days of the last Congress, necessitating a new push this year.

At present, the Treasury Department testified, there are 12 waivers amounting to $47 million in waived duties over the past four years. The waivers cover commodities ranging from Danish butter cookies to Canadian fish and Brazilian textiles.

Under the administration's proposal, the waiver authority would be extended to Oct. 20, by which time the administration hopes to have approval of a new subsidies agreement that would ban all export subsidies for industrial goods and limit the trade advantage for domestic subsidies, including farm supports.

In the meantime, the Treasury Department, instead of arousing the ire of the Europeans and others by trying to collect the duties, has required the posting of bonds or letters of credit to cover the unpaid duties in case Congress rejects the waiver.

The plight of Rep. Hughes' butter cookie maker points to a larger issue in the textile industry, which also claims damage from subsidized imports and is pressing the administration for protectionist concessions in exchange for agreement to the waiver. A proposal to exempt textiles from tariff cuts passed Congress last year but was vetoed by the president. A similar bill has been introduced this year and, if attached to the waiver, also could jeopardize the trade agreements, Strauss has warned.

The trade subcommittee, which appeared to be leaning yesterday toward approval of the waiver, is tentatively scheduled to act on the proposal next Wednesday, although it may wait until the textile accommodation is nailed down, congressional sources said.