In a tough message aimed at protectionists in Congress and overseas, Treasury Secretary James A. Baker III said the United States may consider forming a free-trading "club" of nations along the lines of the recently negotiated pact between the United States and Canada.

Baker said the Reagan administration still intends to seek worldwide liberalization of trade through negotiations to strengthen the General Agreement on Tariffs and Trade (GATT), the compact that has governed international commerce during the postwar period.

But he added, "We can demonstrate a hard-nosed Yankee-trader realism about bargaining: If all nations are not ready {to liberalize trade rules}, we will begin with those that are and build on that success."

Baker's statement, in an article he wrote for The International Economy, a new magazine, appeared designed to show that the Reagan administration is pursuing an activist trade policy and is taking a harder line with countries that restrict the sale of U.S. products.

The Treasury secretary is worried that U.S. lawmakers, who have criticized administration trade policy as too passive, may approve protectionist trade legislation this year and frighten the financial markets.

Baker said one of the benefits of the free-trade treaty signed Saturday by President Reagan and Canadian Prime Minister Brian Mulroney is that it provides "an incentive to other governments" to lower trade barriers.

"If possible," Baker said, "we hope this followup liberalization will occur in the Uruguay Round" of GATT negotiations. "If not, we might be willing to explore a 'market liberalization club' approach, through minilateral arrangements or a series of bilateral agreements."

U.S. trade negotiators have occasionally issued similar warnings in the past in an effort to spur the pace of GATT talks, which tend to move slowly. But such a statement from Baker, the administration's chief economic policymaker, carries more weight.

Moreover, the Treasury secretary's words indicated a somewhat greater receptivity to the idea of negotiating more free-trade pacts with individual countries besides Canada.

In November, U.S. Special Trade Representative Clayton K. Yeutter said that it would be "premature" for the United States to enter into such discussions, even though a number of nations have expressed interest in doing so, because the Canadian-U.S. agreement had just been completed and the GATT negotiations were heating up.

Baker insisted that the Canadian-U.S. accord doesn't undermine GATT, and said he is hoping for "breakthroughs" in the GATT talks on trade disputes involving services, intellectual property and agriculture.

"But many of these beneficial results, if they can be achieved, are years from full fruition," Baker said.

"In the meantime, we need examples of productive government activism that invigorate the bipartisan internationalists in Congress, reawaken businesses and consumers to the gains of open trade, and present possible models for arrangements with other nations," Baker added.