The New York Times failed in the Supreme Court yesterday to win a stay of a New Jersey judge's order to show him the files of a reporter whose investigative articles led to the prosecution of a surgeon on charges of murdering five hospital patients with a potent drug.

Justice Thurgood Marshall denied the newspaper's application for the stay two days after Justice Byron R. White done the same.

Tomorrow in Superior Court in Hackensack, The Time and the reporter, M. A. Farber, must appear before Judge Theodore Trautwein to show cause why they should not be held in contempt for refusing to comply with the order.

If they lose, the could appeal in the new Jersey state courts and possibly bring the case into the Supreme Court once more, although on a different issue - contempt proceedings rather than validity of subpeonas.

The order was issued by another Superior Court judge, William J. Arnold, who is conducting the murder trail of the surgeon, Dr. Mario E. Jascalevich.

During the trial, now in its 20th week, defense lawyer Raymond A. Brown sought Farber's files, including interview notes and tape recordings. This led first to the issuance of subpoenas and then to the order to the newspaper and the reporter to supply the materials on pain of being held in contempt.

On July 6, the New Jersey Supreme Court upheld the order but delayed enforcement to give the paper and Farber an opportunity to seek relief in the U.S. Supreme Court.

In applying here for a stay, they said that the subpoenas, which they had sought to quash, were illegal under state law because they sought irrelevant material and hence were overboard; violated the New Jersey "Shield Law," which is intended to enable reporters to protect news sources, and invaded the press freedom protected by the First Amendment to the Constitution.

Although Judge Arnold was seeking to examine Farber's files in his chambers to determine whether they were necessary to a fair trial, The Times said the obligation of the press to protest its sources required it to withhold the materials from the government, of which the judiciary is a part.

Justice Marshall acknowledged the possibility that forced disclosure, "even to a judge for in camera inspection, will have a deleterious effect on the ability of the news media effectively to gather information in the public interest. . . ."

But, he wrote, the paper and reporter are "plainly not entitled" to a stay in the midst of a trial. They can rise the same arguments in the contempt proceedings, he said. Noting that the order involves in camera in spection, he said, "whether the material will eventually be released to the public is a matter yet to be litigated."

In a similar First Amendment case in California, Sacramento Union reporter John Hammerly was ruled in contempt Tuesday and ordered jailed for refusing to surrender subpoenaed notes and tapes about a murder case.

Superior Court Judge John J. Boskovich in Sacramento said Hammerly will be locked up until he agrees to release the material for an in-chambers review, but he stayed the order for 30 days to allow an appeal.

In the notes and tapes, Edward Gonzales, an unindicted co-conspirator in the slaying of Ellen Delia, told Hammerly he drove a car for her husband and three other men who have been indicted for the killing.