The New York Times asked the Supreme Court yesterday to block a "wholesale invasion" of a reporter's investigative file, saying it contains "source and confidential materials" protected by the First Amendment to the Constitution.

Justice William J. Brennan Jr. is expected to act quickly on an application for a stay filed by The Times and reporter M.A. Farber in the so-called "Dr. X Case" - a New Jersey case involving several murders allegedly committed by use of a powerful muscle-relaxing drug called curare.

The application may set the stage for a major new conflict in the high court between the constitutional guarantees of freedom of the press and the right to a fair trial.

In the trial at issue, the prosecution is now in the 20th week of its effort in Superior Court in Hackensack to prove that while chief of surgery at Riverdell Hospital in Oradell in 1965 and 1966. Dr. Mario E. Jascalevich murdered five surgical patients with curare.

Farber made lengthy inquiries into the deaths, which for years were unexplained. Then, in early 1976, he wrote a series of articles in The Times that prompted the Bergen County prosecutor to reopen an investigation. A grand jury returned a five-count indictment 26 months ago.

The stay application arose form efforts by Jascalevich to subpoena all the information Farber had gathered, including notes of interviews and recordings.

On May 19, relying on an affidavit by Raymond A. Brown, the physician's counsel that the information was necof prosecution witness, trial Judge William J. Arnold certified that it was material to a fair trial and therefore must be produced. This led to issuance by a New York court of subponeas for the papers.

First in New York State and then New Jersey courts, The times and Farber sought to quash the subponeas, arguing that they are impermissibly overboard, that there has been "no showing" of necessity for the materials and that the materials "are protected against forced disclosure by the U.S. Constitution and by the laws of New York and New Jersey."

At a hearing on June 30, Arnold declined to rule on the merits of the effort to quash pending his examination of the materials. He rejected arguments that before examining them, he should rule whether the subponeas and preliminary rulings were invalid on their face.

In the legal battle that followed, Arnold issued an order to The Times and Farber to show cause why they should not be held in contempt for failing to produce the materials.

Last Tuesday, the New Jersey Supreme Court denied a stay requested by the newspaper and the reporter, but allowed them until 5 p.m. today to seek relief from the U.S. Supreme Court.

In yesterday's application, The Times and Farber told Justice Brennan that to allow the subpoenas to be enforced without judicial review would result in a severe limiting of their "ability to gather and report the news, particularly in the area of criminal investigative reporting."

Without judicial review, the application said, the mere issuance of a subpoena makes "the files of the news media . . . instantly subject to perusal for little or no cause," in violation of the First Amendment.

With the trial scheduled to continue "for at least several weeks," granting a stay will cause the physician no irreparable injury, the application said. If he ultimately prevails, he will have "sufficient time to make use of the . . . materials," the paper added.