Interior Secretary James G. Watt exceeded his constitutional authority when he sold five coal leases in North Dakota in defiance of a House Interior Committee directive to postpone the sale, a federal judge ruled here yesterday.

U.S. District Judge Louis Oberdorfer ordered Watt not to issue the leases in a ruling that set the stage for a new constitutional confrontation between the legislative and executive branches of the federal government. The issue might be resolved with the Supreme Court further defining the scope of its landmark June 23 decision striking down the power of Congress to veto actions by the executive branch.

A Justice Department spokesman said late yesterday that the department has decided not to ask the U.S. Court of Appeals for an emergency stay of the ruling.

Oberdorfer's decision was a major setback for Watt, whose coal-leasing program has been under fire from congressional critics who contend that it amounts to a giveaway of public resources.

The decision came a week after the Republican-controlled Senate voted by a two-thirds majority to bar Watt from leasing coal for six months, starting Saturday, the first day of fiscal 1984. The House has passed a stiffer ban. And the two measures were in a conference committee yesterday.

This apparently will invalidate the North Dakota leases until at least next spring. The lease sale sparked controversy because more than three-fourths of the tracts offered by Interior drew no bids. And the other five tracts drew one bid a piece, each at or near the legal minimum.

Watt had no comment on yesterday's ruling, but Interior issued a statement saying, "The issue in this case is a constitutional one rather than a challenge to the coal-leasing program. Its resolution through the courts could benefit all parties."

Watt argues that the Supreme Court decision striking down the legislative veto voided a provision of a federal lands law under which the House Interior Committee could temporarily "withdraw" certain lands from activities that damage them.

Justice Department attorney Robert B. Daniel said that "it is impossible to reconcile the judge's decision" with the Supreme Court's ruling. "We are disturbed because we think the decision is plainly wrong--because the statute and the regulation are plainly unconstitutional after INS v. Chadha."

Oberdorfer, in issuing his injunction, said that the Chadha decision focused on Congress' legislative powers outlined in Article I of the Constitution, not on Congress' power over federal lands, which is outlined in Article IV.

"It is not at all clear" that the provision of the federal lands law is void, Oberdorfer said. "Indeed, it may well be authorized by Article IV . . . . ," he added.

Oberdorfer also cited a 1947 Supreme Court decision, written by former Supreme Court justice Hugo L. Black when Oberdorfer was Black's law clerk, that said that Congress' power over public lands was "without limitation" and that "neither the courts nor the executive agencies could proceed contrary to an act of Congress in this congressional area of national power."