Four Navy men accused of spying for the Soviet Union caused "enormous damage" to national security, Sen. Patrick J. Leahy (D-Vt.), vice chairman of the Senate Select Committee on Intelligence, said in an interview broadcast yesterday.

"Ask yourself this question: What do you think we would have given if we could have gone into the Soviet Union and got people with the kind of access that they had and got them to give that to us?" Leahy said on the "John McLaughlin: One on One" television show.

"We would have considered it one of the biggest coups of this decade," he said.

Retired communications specialist John Anthony Walker Jr.; his brother, retired lieutenant commander Arthur James Walker; his son, Seaman Michael Lance Walker, and his friend, retired senior chief radioman Jerry Alfred Whitworth, face espionage charges. All are being held without bail.

Leahy, who in his role on the intelligence committee has been briefed about the possible implications of the Walker case, declined to specify the nature of the damage the alleged spy ring may have caused.

But, he said, "Consider the fact that the most invulnerable part of our nuclear triad, and the thing that we watch the most on the Soviets, in both instances, is submarines, and just think what they've given away."

Government officials and military analysts have offered varying assessments of the damage that might have been caused by the alleged espionage ring. Sen. David F. Durenberger (R-Minn.), chairman of the intelligence panel, said in an interview last week that the possible passing of information by the men "certainly wasn't helpful" to the United States but that the information "wasn't of such significance that there's any kind of alarm."

Leahy blamed the Navy for failing to uncover the alleged espionage, which law enforcement officials charge has been going on for as long as 20 years.

"How can anybody in the Navy claim they're not negligent in not discovering it or not finding out about it going on during that time?" he asked.

Leahy said Congress should reduce the number of Soviet citizens with diplomatic immunity admitted into the United States. He cited estimates of 1,000 Soviet intelligence agents operating here.

"When you give that number . . . people sort of throw up their hands and think this is some kind of a thing out of a 1950s right-wing fantasy," he said. "What nobody wants to accept . . . is that we are the primary target of Soviet espionage. They've been operating here . . . through detente and not through detente, and they continue to do it and we don't take it seriously enough."

Leahy said, however, that budget increases for counterintelligence operations are beginning to pay off. "We're actually catching some" alleged spies, he said.