President Reagan invoked executive privilege last night to deny access by Senate Judiciary Committee Democrats to internal memos Chief Justice-designate William H. Rehnquist wrote when he was in the Nixon administration Justice Department.

Assistant Attorney General John Bolton made the announcement in the committee room about 8 p.m., just after Rehnquist concluded his second day of testimony at his confirmation hearings. Bolton said the administration was invoking the controversial executive privilege doctrine to protect the confidentiality and candor of the legal advice received by presidents and their assistants.

Democrats, some of whom had left, returned hurriedly upon hearing of the unusual action and reacted angrily. Sen. Edward M. Kennedy (D-Mass.) called it "stonewalling" and Sen. Howard M. Metzenbaum (D-Ohio) charged a "cover-up."

But it was uncertain last night what, if anything, the Democrats could do about the withholding of the memos. Committee sources said they did not have the votes necessary to subpoena the memos and provoke a court battle.

Kennedy told Bolton that the Justice Department was "doing a disservice" to Rehnquist, who had agreed to release of the documents. Kennedy said he could not "reach any other conclusion but that the administration is stonewalling."

Kennedy noted that the period during which Rehnquist served as assistant attorney general in charge of the Justice Department's Office of Legal Counsel -- 1969 to 1971 -- was a time of such controversial activities as Army surveillance of antiwar dissidents, illegal CIA domestic operations and the Huston plan to step up government spying in the United States.

The department's legal counsel is often called upon for opinions -- formal and informal -- on the legality of administration policy. Rehnquist's public defenses of Nixon administration policies on wiretapping, surveillance and mass arrests of antiwar protesters were a source of considerable controversy then. Selected opinions by the legal counsel have been regularly published since 1977.

"Human experience," Bolton told the committee, "teaches that those who expect public dissemination of their remarks may well temper candor with a concern for appearances . . . to the detriment of the decision-making process." He said the privilege was not "being lightly invoked" and that the administration was acting "for the benefit of the Republic," rather than for any particular president.

Sen. Joseph R. Biden Jr. (D-Del.) criticized the department last night for issuing a "blanket exception" and said, "I think you all are making a big mistake."

Metzenbaum told Bolton he thought the action was a "deliberate cover-up . . . . You may try to give it a higher profile," he said, by invoking constitutional arguments of "separation of powers, but that just doesn't fly . . . .

"We are entitled to know what the facts are . . . what's so secret . . . . The president has a right to do it," Metzenbaum said but added that he should not do so on this occasion.

Kennedy indicated last night that he would ask for a committee vote to subpoena the documents. Committee Chairman Strom Thurmond (R-S.C.) said in response, "I consider the matter closed."

Thurmond aide Mark Goodin said the Democrats "are obviously on a fishing expedition, and they need the documents now. They are not interested in a protracted legal fight," which could result in a lengthy delay in the confirmation process.

Executive privilege "is a very murky field," said A.E. Dick Howard, a constitutional law scholar at the University of Virginia. When one branch is battling with another, he said, "courts tend to step aside. The resolution is typically political, not judicial."

The Supreme Court explicitly accepted the validity of "executive privilege" in 1974 in U.S. v. Nixon, when which the justices nonetheless ordered President Richard M. Nixon to turn over White House tapes to the special prosecutor investigating the Watergate scandal. However, the court said in a footnote that its decision did not address the issue of the scope of executive privilege in the face of congressional demands.

Stanley M. Brand, House general counsel during a House committee's 1982-83 battle with the Reagan administration and Environmental Protection Agency Administrator Anne M. Burford over access to Superfund documents, called the assertion of executive privilege "highly offensive."

"In this matter I think the claim of executive privilege is even weaker than in the EPA case because the documents are relevant to the constitutional power of confirmation," he said. While congressional power to investigate is not expressly provided in the Constitution, he added, "here you're talking about a power that the Senate has conferred on it" by the Constitution to give its advice and consent to those nominated by the president