The Reagan administration is considering tough retaliatory sanctions against the Canadian government if it goes ahead with nationalization plans aimed at limiting U.S. investment in Canada.

Actions under consideration range from abandoning the U.S.-Canada auto pact to granting the president emergency powers to eliminate or alter any trade agreement between the two countries.

The actions being considered resulted from a review ordered by the Cabinet at a meeting last month, administration sources said yesterday. At that meeting, the Commerce Department and the U.S. trade representative were asked to come up with a list of possible sanctions.

The Canadian government was notified of the White House study here Tuesday during consultations on Ottawa's "Canadianization" program.

A Canadian Embassy spokesman yesterday acknowledged the meeting with administration officials, but said it would be hard for the White House to design a program to hurt Canada without also hurting the United States. He said the talks were continuing and that he expected them to produce satisfactory results.

"We're letting them know there is serious concern, not just in Congress but also in the administration, and they can't count on the administration to stand between Canada and Congress," Raymond J. Waldmann, assistant secretary of commerce for international economic policy, said in an interview.

The administration has set no timetable for determining what action it might take, Waldmann said, "but we felt it was important to let the Canadians know now."

"There's still an opportunity" for Canada "to look at what they're trying to achieve through these measures and consider the interests of other countries," Waldmann added. "The president is interested in harmony with Canada, yet we have this incident that's in the way."

Canadian Prime Minister Pierre Elliot Trudeau and his Liberal Party were reelected last year on a platform promising increased Canadian control of the economy, especially in energy.

Trudeau has proposed a new energy policy designed to reduce foreign ownership of the oil and gas industry from 72 to 50 percent by 1990 and to make Canada self-sufficient in energy.

Trudeau also has proposed tighter controls on non-energy subsidiaries of foreign companies operating in Canada. U.S. firms have complained that the proposals are aimed primarily at them. Nearly 80 percent of foreign investments in Canada are controlled by U.S. interests.

Besides the administration sanctions, legislation is pending in Congress to force a moratorium on Canadian investment here and stiffen financial requirements for Canadian firms attempting to acquire American companies, Waldmann said. He said the White House is considering throwing its support behind that legislation.

He added that Congress is pressuring the administration to invoke the Mineral Lands Leasing Act, which prohibits a foreign company from leasing federal land for mineral development unless that country reciprocates. The secretary of the interior can declare Canada "non-reciprocal."

Waldmann said the administration was not trying to act as a big brother telling Canada what to do, but he acknowledged that it was "finger-twisting."

The most far-reaching option being considered by the White House would allow the government to initiate an investigation under Section 301 of the Trade Act of 1974. That, after a government determination, could give the president broad powers of retaliation.

The determination that Canada violated an agreement or conducted unjustifiable trade practices must be backed by evidence, which the Commerce Department is gathering. Several weeks ago Commerce mailed to the executives of the Fortune 500 companies a questionnaire asking them what problems they have had with the Canadian government, Waldmann said.

Another issue clouded by the administration's review is the U.S.-Canadian auto pact, which the Canadians have said they want renegotiated to provide more benefits for their companies, which are mostly divisions of U.S. firms.