The House Energy and Commerce Committee voted 23 to 19 yesterday to cite Interior Secretary James G. Watt for contempt of Congress for refusing to turn over subpoenaed documents.

The vote, which could lead to a major constitutional confrontation between the House and the Reagan administration, came after Democratic critics accused Watt of a pattern of obstructionism in an investigation that began last July.

If the contempt citation goes to the House floor and the House concurs in it, Watt could be brought to trial in a U.S. District Court or, in a seldom-used alternative, arrested and tried by the House itself. The penalty on conviction could be a fine of up to $1,000 and a jail sentence of up to 12 months.

Researchers said yesterday they could find no modern example in which a Cabinet officer was cited for contempt by a house of Congress. The House Intelligence Committee voted to cite then-secretary of state Henry A. Kissinger in 1975, but the case was settled before it reached the House floor. In two more recent cases, citations voted by subcommittees never reached the full committees.

The Watt vote was along party lines except for one Democrat who voted against the citation and one Republican who voted for it.

Committee Democrats charged that Watt withheld many documents for weeks and has still refused to turn over between seven and 11 documents related to an administration study of Canada's oil policy.

Republicans retorted that Watt was being assailed for strictly partisan reasons because of his controversial policies at the Interior Department or pilloried by foes who dislike his manner.

In a statement later, Interior spokesman Douglas Baldwin said only seven documents are still in dispute, all of them protected on orders of President Reagan. The case of executive privilege probably will be resolved by the Supreme Court, he said.

Last-minute efforts to settle the Watt case failed. Committee Chairman John D. Dingell (D-Mich.) said that early this week a conference with White House counsel Fred Fielding ended unsuccessfully when Fielding agreed only to brief committee members on the documents, not to turn them over to all the members. Dingell said he refused the deal. But he said the committee would remain open to further offers of compromise with the executive branch.

The administration contends the remaining documents fall under executive privilege because of foreign policy considerations or because they represent internal discussions at the Cabinet level.

Because the Justice Department drafted that line of defense, it would be in the unusual position--if the full House acts--of being instructed to prosecute Watt for abiding by the very advice it gave him. A Justice spokesman had no comment on what the department would do if the House votes contempt.

If the Justice Department declined to prosecute, the committee majority would attempt to have Watt tried on the House floor after being arrested by the House sergeant-at-arms, committee sources said.

Another alternative would be for congressional counsel to ask a three-judge federal court for appointment of a special prosecutor to take charge of the case.

The affair began last year when the committee's oversight subcommittee sought details on an administration inquiry into Canada's new energy policy, which some American firms think is discriminatory against U.S. oil companies.

Watt originally was asked to answer a series of questions and to turn over more than 200 documents. In response to continuing committee pressure, Watt has submitted all but seven or 11 of them (the exact number is disputed). They are said to involve State Department cables or accounts of Cabinet deliberations that the White House insists are protected by executive privilege.

Dingell said yesterday the committee sought the documents "so we could know how well the law was working, how well he was doing his job and what, if anything, needed to be changed. His response was to huggermugger the documents that would have answered those questions. When we pushed our point, he ran to the president."

The committee's ranking minority member, Rep. James T. Broyhill (R-N.C.), insisted that Watt had made a "good faith" effort to comply with a subcommittee subpoena and is now merely carrying out presidential instructions.

"It would be unseemly to have the sergeant-at-arms arrest a Cabinet member" who was merely carrying out presidential orders, he said.