The Supreme Court refused yesterday to review the convictions of The New York Times and Myron A. Farber for civil and criminal contempt in disobeying a judge's order that the reporter surrender his full investigative file on a New Jersey murder case.

As is customary, the justices gave no reason for denying a petition for review of a decision by a lower tribunal - in this case, a 5-to-2 ruling in which the New Jersey Supreme Court upheld the convictions.

All that is certain is that four of the nine justices - the minimum number needed - didn't vote to grant the petition. In a preliminary phase of the case, only one of them, Justice Thurgood Marshall, said that he would grant the petition. Justice William J. Brennan Jr., without explanation, has disqualified himself throughout.

A denial of a petition for review can be based on any of numerous reasons, none of which was disclosed in this case, and implies nothing about the justices' views of the merits.

The result in the Farber case is to leave the New Jersey decision standing in that state. It also leaves for a possible future Supreme Court ruling questions about the "shield" laws enacted by 26 states to enable reporters to protect confidential sources, about "balancing" of freedom of the press under the First Amendment and the right to a fair trial under the Sixth Amendment, and about other major issues.

Reporter Farber's file contained his notes, tapes and other materials from an investigation into the unexplained deaths of five patients at a hospital in Oradell, N.J., where Dr. Mario E. Jascalevich was chief of surgery.

Following publication of Farber's stories in The Times, prosecutors reopened their investigation, which resulted in the indictment of Jascalevich on charges of murdering three of the patients with a muscle-relaxing drug. A jury acquitted him last month.

Farber was jailed for 39 days during the trial but was released after the acquittal. Meanwhile, The Times had paid fines totaling $285,000.

In New York City yesterday, Farber, expressing regret at the Supreme Court's denial of his and his newspaper's petition for review, said:

"The issues presented in our case - the ability of reporters to preserve the confidentiality of their sources and, at the very least, to receive a full and fair hearing - have arisen throughout the country with increasing frequency in recent years, and with differing lower-court opinions. They will undoubtedly arise again and will ultimately have to be addressed by the nation's highest court.

"I trust that when the time comes, the Supreme Court will give full recognition to the vital role played in this democracy by a responsible and independent press.

"I, and The Times, have tried in this matter to uphold the Constitution, not in any parochial sense for our own benefit, but in the public interest. We can do no more, and, certainly, we should have done no less."

Much of the uncertainty left by yesterday's action arises from a 1972 decision in a case called Branzburg vs. Hayes.

In that opinion for the Supreme Court, Justice Byron R. White seemingly invited the states to pass shield laws, saying that they afford journalists "a privilege to refuse to disclose, in any legal proceeding" or to "any court," the "source . . . from or through whom any information was procured."

Accepting the invitation, New Jersey enacted a strong shield law. But efforts by Farber and The Times to invoke it failed after Trial Judge William J. Arnold, at request of the Jascalevich defense, ordered them to surrender the file.

The state high court, in the first major test of the law, intimated that it didn't apply to a reported who is also an investigator, although a dissenter said, "All good reporting must be investigative."

In the Farber case, the court said, the law must yield to the Sixth Amendment's provision for compulsory production of materials needed by a defendant. But, it said, the law will protect journalists hereafter.

The Branzburg case involved a reporter's claim, rejected by the Supreme Court, that he had a First Amendment privilege to disobey a subpoena to testify before a grand jury.

But in a separate opinion, Justice Lewis F. Powell Jr. wrote than such a claim "should be judged on its facts by the striking of a proper balance between freedom of the press and the obligation of all citizens to give relevant testimony with respect to criminal conduct."

The highest courts of some states, including Virginia, Vermont and Wisconsin, have held that just such a balancing is required, Floyd Abrams, attorney for Farber and The Times, pointed out. New Jersey's did not.

Farber and The Times sought but were denied a hearing before Judge Arnold on his demand for the file. This deprived them of their right to due process of law, they argued. The New Jersey Supreme Court disagreed.

In opposing U.S. Supreme Court review, Jascalevich's attorney, Raymond A. Brown, said that Farber and The Times were seeking "to be above the law and accountable only to themselves."

In another First Amendment case, the court yesterday let stand a Louisiana appeals court ruling that Brian King couldn't combat a crime wave in his Alexandria, La., laundry with a long-running bulletin-board display of hidden-camera photos of Michael G. Norris "Caught in the Act" of burglary and a text giving Norris's name and recounting his arrest and punishment.

The ruling said that such activity by an individual and the publication by a newspaper of truthful information about a criminal are on different First Amendment footings.