Attorney General William French Smith has moved to cut the legs out from under House Democrats in one of their more open efforts to "discipline" Smith's fellow Cabinet officer, Interior Secretary James G. Watt.

The attorney general, in a move that one House Democratic aide described as the beginning of "a real power struggle," has informed House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) that the Justice Department will not defend Congress in a lawsuit over use of a Montana wilderness area.

Smith's rebuff could force Congress to hire its own lawyers to defend the case, which grew out of a House Interior Committee attempt to stop Watt from opening the Bob Marshall Wilderness Area to oil and gas leasing.

Almost 350 mineral-leasing applications are pending in Bob Marshall, a collection of three adjoining wilderness areas comprising 1.5 million acres on the eastern slope of the Rockies.

The episode represents the latest twist in a rather messy showdown between Watt and the Interior Committee. The controversy began in May when the committee, on a strict party-line vote, used a little-known emergency provision of the Federal Land Policy and Management Act to place a three-year moratorium on mineral leasing in the wilderness area.

The congressional action was challenged almost immediately in federal court by western pro-development groups, including the Mountain States Legal Foundation, which Watt served as executive director until he joined the Cabinet.

The committee's action, viewed widely as a rebuke to Watt by angry Interior Committee Democrats, came at the height of almost open warfare between Watt and the committee that oversees his department.

The committee chairman, Rep. Morris K. Udall (D-Ariz.), acknowledged at the time that the action was "novel." But Udall said the committee was miffed with Watt's tactics and "this is the price he is paying for this kind of confrontational administration."

Another key committee member, Rep. John Seiberling (D-Ohio), said the action was designed to "send a message to the other end of Pennsylvania Avenue."

The message that came back, however, was from the attorney general. Smith wrote O'Neill that the Justice Department not only would decline to defend the congressional position but would urge U.S. District Court in Billings, Mont., to rule that the law's emergency provision was unconstitutional.

Watt, who has spent the past week in Alaska, said yesterday he was unaware of Smith's letter. "But I'm not surprised," the secretary said, adding that his Interior Department attorneys had offered the same opinion. Watt said he has tried to stay out of the legal controversy because the lawsuits technically are filed against him as the implementer of the congressional mandate.

Udall, too, seemed unsurprised, and said the showdown illustrated his original point that "policy by confrontation" can ensnarl both sides.

Udall said he found the latest showdown ironic because Watt had backed down after the controversial congressional move. He said Watt came to his office and said he would go along reluctantly with the Montana moratorium because he didn't want to threaten the rest of his mineral-leasing program through more confrontations.

Udall said he was worried now that the constitutional showdown might overshadow the fate of the wilderness area. He did not rule out bringing in congressional lawyers to defend his committee's unprotected position in court, but he said little could be done before Congress returns to work next month.

Meanwhile, the prime mover behind the committee action, Rep. Pat Williams (D-Mont.), called Smith's action "a threat to Congress' authority to tell the bureaucrats what to do." Williams said he would ask Congress to intervene with its own lawyers.