The trial judge in the espionage court-martial of Marine Sgt. Clayton J. Lonetree refused yesterday to grant the defense access to naval investigators' personal files and training manuals, despite what defense lawyers claimed is the possibility of "pervasive misconduct" by the investigators.

The judge, Navy Capt. Philip F. Roberts, accused the defense of wanting a "gigantic fishing expedition" to secure documents that might not be relevant to the case.

But in heated exchanges on the third day of the trial at Quantico Marine Base, defense lawyer Michael V. Stuhff said the defense is being "handcuffed," and he accused the court of an abuse of discretion.

"We are not on a fishing expedition," Stuhff said. "We have reasonable cause to believe that naval investigative procedures are coercive and unlawful."

The military lawyers prosecuting Lonetree told the court that the defense will have the chance to examine Naval Investigative Service witnesses when they appear. Marine Corps attorney Maj. David L. Beck said, "If the defense wants to prove alleged threats and coercion, they will be free to try to do so."

Lonetree, 25, an American Indian from Minnesota, is charged with espionage, multiple counts of conspiracy to commit espionage, failing to obey regulations and wrongfully disclosing the identifications of U.S. citizens. If convicted, he could be sentenced to life imprisonment. His lawyers say they will enter formal pleas of not guilty to all charges.

In arguing their motions, Lonetree's lawyers emphasized that they need more information to prove that incriminating statements Lonetree made to investigators detailing his contacts with Soviet citizens while he was a guard at the U.S. Embassy in Moscow were not voluntary.

Stuhff told the court that in pretrial hearings there already have been admissions of illegal conduct from two of the investigators involved in the case.

Stuhff said the personal files may reveal "other actions, coercion of witnesses, fabrication of evidence and intimidation of defense witnesses." Without access to such documents, the hearing would dissolve into a "sham and a farce," he said.

The judge also rejected a defense motion for production of a classified report by Texas billionaire H. Ross Perot on security at U.S. embassies. Stuhff said Perot told him that the report, which was ordered by President Reagan, revealed "a sloppy, negligent attitude to security" at the embassy in Moscow.

Stuhff added, "This whole case involved the intent and motivation of Sgt. Lonetree. Relevant to his intent is the entire attitude of the authorities toward security at the Moscow embassy and the attitudes of other officials including the ambassador."

The trial judge rejected a motion by defense lawyer William Kunstler to dismiss the case on grounds of the investigators' incompetence.