The Reagan administration has begun cautiously waving the threat of retaliatory action against Canada for measures that have put U.S. energy firms at a disadvantage there and precipitated a rash of attempts by Canadian firms to take over holdings of U.S. energy companies.

Treasury Undersecretary Beryl W. Sprinkel told a House Energy and Commerce subcommittee that the administration will ask the Interior Department to expedite a review of whether Canadian firms are entitled to mineral, oil and gas leases on U.S. federal lands or whether they should be cut off because of steps Canada has taken to nationalize energy production there.

"Canadian measures are causing strong pressures" to trigger a provision of the Mineral Lands Leasing Act that would deny Canadian firms access to leases on federally controlled lands, Assistant Secretary of State Robert D. Hormats told the same hearing. Foreign firms are allowed to explore for and extract minerals under leases on U.S. federal land as long as U.S. companies have the same right in the foreign firms' countries.

Those recent Canadian actions include development of a National Energy Policy that offers energy exploration incentives to firms based on their percentage of Canadian ownership and limits the ability of U.S. firms to develop oil and gas in Canadian federal lands, and the Foreign Investment Review Agency, which screens new foreign investments in Canada to determine their "benefit" to Canada.

Deputy U.S. Trade Representative David R. MacDonald said he believes Canadian actions are not reciprocal under the provisions of the Mineral Lands Leasing Act, but added that the determination will have to be made on legal grounds.

"If restrictions on transfer or forced sales are imposed on U.S. interests as a result of the action of a foreign government, then similar restrictions should be imposed on companies controlled by foreign stockholders doing business in the U.S.," House Energy and Commerce subcommittee Chairmen John D. Dingell (D-Mich.), and Rep. Tim Wirth (D-Colo.), wrote Interior Secretary James Watt.

The two congressmen urged Watt to speed up his review of reciprocity, noting that it has been under way since May with no results.

"Canada's policies towards foreign investment have since the mid-1970s been moving in a restrictive and discriminatory direction," Hormats said. "The United States is seriously concerned about this situation," he said. But Hormats and Sprinkel both noted that the United States is reluctant to take retaliatory action.

Administration officials also noted diplomatic steps being taken in an attempt to modify the Canadian position and said that discussions of the problem with Canada have reached the Cabinet level. U.S. concerns about Canada's energy measures are expected to be on the agenda today when President Reagan meets with Canadian Prime Minister Pierre Trudeau.

"I certainly wouldn't be surprised if it was brought up in a rather pointed fashion," said Sprinkel.