The Supreme Court yesterday held unconstitutional the new court system that was created by Congress in 1978 to streamline the handling of thousands of bankruptcy cases.

Acting at a time of record bankruptcy filings, the court said Congress gave insufficient protection to the independence of the powerful bankruptcy judgeships by denying them life tenure and protection against salary reductions.

The court took clear steps to try to limit the damage of yesterday's decision, saying it would not be retroactive and would be stayed until October. It is nevertheless expected to raise legal questions about all current federal bankruptcy proceedings.

In response, Rep. Peter W. Rodino Jr. (D-N.J.), chairman of the House Judiciary Committee, said he will begin hearings in mid-July on legislation to give the bankruptcy judges the protections the court found necessary yesterday.

The 1978 revisions replaced a system of "bankruptcy referees" with extremely limited jurisdiction. They could make decisions pertaining, for example, to assets in the possession of the company declaring bankruptcy but not decisions relating to assets held by a third party. For those matters, the parties would spend months haggling over jurisdiction or would go one step up, to U.S. District Courts, for rulings.

The revisions created bankruptcy judges with almost all the powers of the district court, including the power to conduct trials by jury, and centralized all bankruptcy-related functions in them.

The new judges thus were given much of the authority exercised by district judges but not the independence the Constitution provides for district judges to protect them from the pressures that come with such authority. Although district judges are appointed for life terms, for example, the bankruptcy judges were allowed 14-year terms. Although district judges may be removed only by impeachment, a vote by a panel of federal judges in each circuit can remove a bankruptcy judge under the new act.

The district court ruled the 1978 act unconstitutional for that reason in a case stemming from a January 1980 bankruptcy petition filed by Northern Pipeline Construction Co. As part of its reorganization, Northern sought damages from Marathon Pipeline Co. over alleged breaches of contract involving a pipeline construction project in Kentucky. Marathon sought the district court ruling, contending that the new bankruptcy court lacked proper constitutional authority to consider the breach of contract matter.

Justice William J. Brennan Jr., joined by Justices Thurgood Marshall, Harry A. Blackmun and John Paul Stevens, wrote the main opinion yesterday, saying that the new system represented an "unwarranted encroachment" on the independence of the judiciary.

Justice William H. Rehnquist, joined by Justice Sandra Day O'Connor, said that the exercise of bankruptcy court power in the Marathon case was unconstitutional but that the court had gone too far in declaring the entire act impermissible.

Justice Byron R. White, with Chief Justice Warren E. Burger and Justice Lewis F. Powell, dissented. Burger wrote a separate dissent as well.

In other action yesterday, the court:

* Voted 6 to 3 in a significant ruling for the health insurance industry to expose a common technique for settling medical insurance claims to the threat of antitrust suits. The court ruled that an insurance exemption in antitrust law does not cover "peer review committees," through which doctors give insurers free advice on whether a peer's services and charges were necessary and reasonable.

* Left intact a ruling that right-to-work laws, adopted by 20 states, including Virginia, cannot be enforced on federally controlled lands within the states' borders.

* Ruled 6 to 3 in a sex bias case against Ford Motor Co. that an employer who has discriminated against a job applicant can minimize his penalty by making a belated job offer.

* Refused to revive an $11 million lawsuit against NBC that blamed TV violence for the sexual assault of a 9-year-old girl.