The Reagan administration warned yesterday it may take retaliatory action against Canada, the United States' largest trading partner, if discussions to eliminate discriminatory trade and investment practices there aren't successful.

"This administration has not and will not sit by idly and watch our companies and businessmen bear the brunt of costly and inequitable restrictions in the Canadian market," Assistant Commerce Secretary Raymond J. Waldmann told the Senate Foreign Relations subcommittee on international economic policy.

"I might also point out that should our bilateral or multilateral efforts to eliminate discriminatory Canadian practices fail, then we will proceed to explore the merits of pursuing unilateral measures available under U.S. law," such as section 301 of the Trade Act of 1974, Waldmann said. That provision allows the government to initiate an investigation that could give the president broad powers of retaliation.

"We must, of course, proceed cautiously in this regard, first to ensure that we do not do damage to U.S. interests in taking such action, and second to ensure that we do not do harm to the international trading system," Waldmann continued. "Nevertheless, there should be no doubt that we will use all appropriate legal means to defend our interests."

The administration last fall first publicly warned Canada about its nationalization plans aimed at limiting U.S. investment and which threaten to hurt U.S. businessmen there. Since then, however, rhetoric from Washington had been less strident and some officials said they hoped that a less heated debate could bring results.

Last fall, the administration considered actions ranging from abandoning the U.S.-Canada auto pact to granting the president powers under Section 301 to eliminate or alter any trade agreement between the two countries.

Since then, talks between Canadian and American officials to reduce or prevent discriminatory barriers to trade and investment have not resulted in significant changes in the offending policies, although there is some indication some Canadian programs have been postponed, Waldmann said.

Waldmann's statements were contained in written testimony submitted to the subcommittee that had been omitted during his verbal remarks. However, after the hearing Waldmann said he stands by the statement and so does the administration which approved it. The passages were omitted for brevity, he said.

The administration already has taken some of its complaints against Canada to multilateral forums such as the General Agreement on Tariffs and Trade, where decisions are pending.

Canadian trade officials here found Waldmann's statements "strange when we're involved in a GATT consultation to resolve the complaints about our investment practice," a Canadian embassy spokesman said. "We are committed to seeing the GATT consultation and any GATT action through. At that point the Commerce Department would start talking about applying any unilateral actions."

Waldmann and other administration officials yesterday complained about practices such as Canada's Foreign Investment Review Agency, which screens foreign direct investment and imposes obligations such as legally binding export quotas, import ceilings and other requirements.