Interior Secretary James G. Watt -- invoking a presidential claim of executive privilege -- refused yesterday to turn over 31 documents that a House subcommittee had subpoenaed, opening himself to the possibility of being held in contempt of Congress.

It was the Reagan administration's first claim of executive privilege. Instead of producing the documents demanded by the subcommittee's subpoena, Watt presented a memo to him from President Reagan claiming executive privilege for the materials, which have to do with a trade dispute with Canada.

"It is my decision that you should not release these documents, since they either deal with sensitive foreign policy negotiations now in process or constitute materials prepared for the Cabinet as part of the Executive Branch deliberative process through which recommendations are made to me," Reagan wrote. Under that conclusion, Reagan wrote, "I am compelled to assert executive privilege." The memo ordered Watt not to produce the documents.

Watt told the committee the president had seen the documents for the first time on Tuesday when they were presented to him for a decision on whether to claim executive privilege.

Subcommittee Chairman Rep. John D. Dingell (D-Mich.) stopped short of threatening to move to have Watt held in contempt of Congress. But he indicated that Watt may be cited if no compromise can be worked out and the documents are not produced.

"The patience of the committee is wearing thin," Dingell said. Subcommittee counsel Kathryn Seddon also cautioned an assistant attorney general who acted as an attorney to Watt that the Justice Department might face a conflict of interest in advising Watt beyond a certain point. Justice, she explained, would be called upon to prosecute if Watt were charged with contempt.

In refusing to hand over the documents, the administration relied on several arguments, including the sensitive nature of the papers and a claim that delivery of the documents would disrupt the deliberative process leading to a major decision.

White House spokesman Larry Speakes said the heart of the matter is that the documents deal with advice from Cabinet members to the president, which Speakes said should never be revealed. He also said some of the documents were classified.

The documents in question have to do with the issue of whether retaliatory provisions of the Mineral Lands Leasing Act should be invoked against Canada. Under Canada's energy plan, which is designed to produce greater Canadian control of that country's natural resources, the government has implemented actions that American natural resource companies say discriminate against them. The documents withheld include classified telegrams from the U.S. embassy in Canada, memos from U.S. trade representative William Brock and numerous internal Interior Department memos.

In the wave of concern that followed takeovers and threatened takeovers of American companies or their Canadian subsidiaries, several members of Congress and others suggested that provisions of the leasing act might be invoked that allow Watt to deny federal oil and gas leases to countries that are found not to provide equal opportunities to American companies.

Watt has made no decision on whether to use that power.

The subcommittee had requested a number of documents to determine whether Watt was violating the law by not invoking the reciprocity provisions, Dingell said. After several months of back and forth about what Watt would do and what documents would be produced, the subcommittee voted the subpoena, he said.

Watt asserted that the Interior Department has been cooperative with the subcommittee, producing strong disagreement from Dingell.

Rep. Albert Gore (D-Tenn.) called the claim of executive privilege in this case "absolutely ridiculous."

"There will be a sharp conflict between the Congress and the Executive Branch that is totally unnecessary."