The Supreme Court, rejecting a vehement claim of congressional immunity, yesterday left a group of congressional staff members open to a multimillion-dollar civil damages trial over the way they conducted a 1978 congressional investigation.

Congressional leaders argued against the action, fearing it will leave other congressional investigators in the future open to suits and possible harassment.

The justices, without comment, refused to review a lower court ruling that made a suit and trial possible against staffers of the House Select Committee on Aging. An insurance agent in Baltimore charged them with improperly using concealed cameras and concealed identities to investigate alleged abuses in the sale of health insurance to the elderly. He also sued an ABC-TV producer who participated in the undercover probe and broadcast portions of it.

Yesterday's action by the Supreme Court came over the strong objections of congressional lawyers and House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.), who said it would severely compromise the immunity that shields congressional business from such lawsuits and have a chilling impact on future investigations.

The lower court rulings, O'Neill had told the justices, will have "a dramatic adverse effect on the congressional investigative process."

The Supreme Court's unwillingness to intervene yesterday does not bar its review of the case at some later date and does not give the court's formal approval to the suit. It does mean, however, that the staffers will have to answer questions under oath about their actions, which is exactly what they say the immunity is supposed to prevent. They now fear harassing suits, as well as serious suits, from future objects of congressional probes.

Yesterday's case, Holton vs. Benford, stemmed from a committee inquiry into private insurance sold to the elderly as a supplement to Medicare. One committee staffer posing as a trainee obtained employment with George H. Benford's insurance agency in Maryland. The committee also employed two older women to pose as potential customers.

They then got Benford to give a sales pitch at the home of one of the women, and without his knowledge, allowed ABC to tape the conversation.

Benford sued all of them, including chief investigator David L. Holton, for more than $26 million, charging them with violating both state and federal anti-eavesdropping laws.

In Baltimore, U.S. District Court Judge Edward S. Northrup, who was later affirmed by the 4th U.S. Circuit Court of Appeals, ruled that the congressional defendants were not absolutely immune under the Constitution as they claimed and allowed the case to go to trial for consideration of whether they acted outside their official authority in the insurance investigation.

The Constitution's "speech and debate" clause says that senators and representatives "shall not be questioned in any other place" for their actions as members of Congress. Despite that language, however, the scope of their immunity from suits has not been clearly defined. The same is true for the presidency. The president's immunity from suits is currently being considered by the justices.

In other action yesterday:

* The court said it would consider whether the First Amendment protects the sale of pictures showing children engaged in sexual acts. The court will review New York's "kiddie-porn" law, which was struck down largely because it banned commercial distribution of all such pictures, whether legally obscene or not.

The New York Court of Appeals held, in New York vs. Paul Ira Ferber, that such a sweeping ban was too broad, and could be applied to "films and portrayals of nonobscene adolescent sex solely on the basis" of content. "Those who present plays, films and books portraying adolescents cannot be singled out for punishment simply because they deal with adolescent sex in a realistic but nonobscene manner," the New York court ruled.

Nineteen other states have laws like New York's, and others are considering "kiddie-porn" legislation designed to stifle what has become a lucrative commercial industry.

* The court said it would consider, once again, the circumstances under which judges can close trials to the press and public. Yesterday's case stems from a challenge by The Boston Globe to a Massachusetts law requiring closed rape trials when the victim is under 18.

The Massachusetts courts upheld the law's constitutionality even after the Supreme Court, in 1980, said that most trials should be open. The 1980 ruling, in a case involving Richmond Newspapers, Inc., failed to specify valid reasons for closure, however.

* The court agreed to consider whether peer review systems, committees of doctors used by insurance companies to evaluate claims, are subject to federal antitrust laws. Two cases accepted yesterday, New York Chiropractic Association vs. Pireno and Union Labor Life Insurance Co. vs. Pireno, originated with a chiropractor's suit charging that peer review of his claims breached antitrust laws.