The Supreme Court ruled yesterday that the Constitution does not shield a member of Congress from a lawsuit if he libels someone in a press release or newsletter - even though the defamatory statement may have been made originally in the House of Senate.

Libelous remarks by members in phone calls to executive agencies and in broadcast interviews also are not protected, the court held, 8 to 1, in a case that began with one of the "Golden Fleece" awards made monthly by Sen. William Proxmire (D-Wis.).

The decision turned on the clause of the Constitution saying that "for any speech or debate" a senator or representative "shall not be questioned in any other place" but the Senate or House.

The immunity provided by the clause is not expanded by "such helpful facilities" on Capitol Hill as recording studios and postal franking privileges, Chief Justice Warren E. Burger wrote in the opinion for the court. The franking privileges commonly are used to send out the Congressional Record, among other materials.

Whether the press is liable when it publishes or broadcasts defamatory statements by legislators was not an issue in this case.

The ruling was a defeat for the leadership on Capitol Hill. Separate briefs on behalf of the House and Senate, each signed by the top Democratic and Republican officers, had been filed in Proxmire's behalf. Both briefs expressed concern about decisions in the last several years that, as the leaders portrayed them, improperly intrude on the right of legislators to communicate with the public.

In a statement yesterday, House Speaker Thomas T. (Tip) O'Neill Jr. (D-Mass.) agreed with the dissenting opinion, in which Justice William J. Brennan Jr. said that public criticism by legislators of governmental expenditures is a legislative act protected by the Constitution.

"It is my opinion that it is important for members of Congress to feel free in communicating with their constituents," O'Neill said.

Proxmire said, "The Golden Fleece will go on . . . I will strive to be just as emphatic, vivid, and, if possible, humorous in my denunciation of waste as I can be."

By removing the immunity provided Proxmire by the clause, the decision opened him and an aide, Morton Schwartz, to a libel suit brought by scientist Roland R. Hutchinson. The court having done this, Proxmire said, "I feel no constraint - none - in continuing the fleece awards . . ."

Proxmire started the awards in 1975 to publicize what he deemed to be the most egregious examples of wasteful governmental spending. The second one involved Hutchinson, then research director of a Michigan state mental hospital in Kalamazoo and an adjunct professor at Western Michigan University.

Over the preceding seven years, he had received more than $500,000 from the National Science Foundation, Office of Naval Research and National Aeronautics and Space Administration, mainly to help select crewmen for submarines and spacecraft. It was the agencies to which Proxmire awarded the fleece.

Hutchinson was trying to find an objective measure of aggression, focusing on certain animal behavior, such as the clenching of jaws under stress.

After research by legislative assistant Schwartz, Proxmire made a Senate speech about the funding of "this nonsence," which "should make the taxpayers as well as his monkeys grind their teeth." He found in Hutchinson's study of "jaw-grinding and biting by angry or hard-drinking monkeys" a "transparent worthlessness," yet "the good doctor has made a fortune," he said.

An advance news release incorporating the speech went to 275 members of the press here and abroad. Later, Proxmire sent out 100,000 copies of a newsletter referring to the speech. He also appeared on the Mike Douglas Show on television, but without naming Hutchinson.

The studies then were dropped. "No more monkey business," Proxmire said in a 1976 newsletter. Schwartz, it turned out, had phoned the agencies.

Hutchinson sued for libel, alleging humiliation, extreme mental anguish, physical pain and loss of income. A trial court dismissed the suit, holding in part that speech-or-debate protected the speech and that the "informing function" of Congress and the franking privilege protected the mailed materials and the TV interview.

The 7th U.S. Circuit Court of Appeals affirmed, although it held that the First Amendment protected the broadcast. It was reversed yesterday.

The clause protected the speech because it was essential to Senate deliberations, the chief justice wrote. But there was no constitutional protection "for defamatory statements scattered far and wide by mail, press, and the electronic media," Burger said. As for the "informing function," he said, it was intended for Congress, not the public.

Justice Potter Stewart disagreed only as to phone calls to agency officials. Justice Brennan recalled his dissent, in a 1971 case, in which he wrote that the court was so restricting speech or debate "as to endanger the continued performance of legislative tasks . . . vital to the workings of our democratic system." CAPTION: Picture, SEN. WILLIAM PROXMIRE . . . 'Golden Fleece will go on'