Big changes are in the works among American negotiators handling some of the major trade talks of 1986.

Peter O. Murphy, now deputy U.S. trade representative in Geneva, where he holds the rank of ambassador, is moving back to Washington to handle the critical beginnings of talks with Canada that could lead to a free-trade zone between the United States and its northern neighbor. Murphy will also be responsible for the United States' always ticklish trade relations with the neighbor to the south, Mexico.

Murphy, 37, will be replaced in Geneva by Michael A. Samuels, former head of the international division of the U.S. Chamber of Commerce. Samuels ran President Reagan's transition team in international trade and was ambassador to Sierra Leone in the Ford administration.

Samuels will be dealing with the start of a new round of global trade talks, a major Reagan administration priority. The United States wants to strengthen and expand the authority of the General Agreement on Tariffs and Trade (GATT), the Geneva-based international organization that sets the rules for world trade, in an effort to roll back the United States' record trade deficit, which is expected to reach $150 billion.

The Canadian trade talks are another administration priority. They grew out of President Reagan's "Shamrock summit" in Quebec last March with Prime Minister Brian Mulroney, who gave his formal approval three months ago to start the talks. Reagan asked Congress earlier this month for permission to begin negotiations, with this spring as the deadline for a congressional decision.

The two countries already are the world's largest trading partners, exchanging $120 billion of commerce annually.

William Merkin will be Murphy's aide in his dealings with Canada. Tim Bennett will handle Mexican trade relations, which also are in a period of flux because the southern neighbor has decided to join GATT.

The trade talks are major news in Canada, where there is continuing concern that ending trade barriers could mean tht the United States, with its population of more than 200 million and its greater degree of technological sophistication, would completely overwhelm the 25 million Canadians culturally and economically.

While the idea of free trade with Canada has not imprinted itself as forcefully on the American psyche as it has in Canada, major U.S. industries are staking out positions for the talks.

One of the most visible and highly politicized problems between the two countries are charges by American lumbermen that Canada unfairly subsidizes its timber industry, allowing Canadian wood to take a major share of the U.S. market. The International Trade Commission and Department of Commerce investigated similar charges three years ago and gave the Canadian industry a clean bill of health. Now, the Americans are tackling the issue through Congress, seeking a change in the law governing subsidies to make government-set low prices for a natural resource such as timber an unfair trade practice.

The issue complicates the start of free-trade talks since some key senators, including Finance Committee Chairman Bob Packwood (R-Ore.) and committee member Max Baucus (D-Mont.) come from lumber-producing states and have expressed great interest in the U.S. industry's complaint.

U.S. Trade Representative Clayton Yeutter promised to hold talks with his Canadian counterparts in an effort to solve the lumber issue before spring, the deadline for Congress to veto the trade talks.

Formal negotiations on lumber have not started. But Yeutter and Deputy U.S. Trade Representative Alan Woods visited Canada the week before Christmas for informal talks.

Undoubtedly, the issue will come up again in mid-January when Yeutter is the host for the periodic meetings of trade ministers from the United States, Canada, Japan and the European Community. This "quadrilateral" meeting will be Jan. 16 in San Diego.

While formal sesssions involve all four ministers, there is likely to be time around the swimming pool of the Del Coronado Hotel for Yeutter and Canadian Trade Minister James Kelleher to talk lumber.

In addition to the formal sessions, functions for the trade ministers will be held by Sen. Pete Wilson (R-Calif.), a former mayor of San Diego, and California Gov. George Deukmejian (R), who calls California "a world within a state" because its gross national product is greater than all but six nations.