Ronald W. Pelton, a former $24,500-a-year National Security Agency employe whose trial on charges of selling "extremely sensitive classified information" to the Soviet Union began with jury selection this week, compromised a costly, long-running and highly successful U.S. operation that used sophisticated technology to intercept Soviet communications, according to sources familiar with the case.

The Soviets uncovered the U.S. operation, which involved the use of American submarines, after debriefing Pelton during two extended sessions in Vienna, the sources said. The U.S. government subsequently discovered physical evidence that the operation was compromised. U.S. intelligence officials are convinced the high-technology device used in the operation was retrieved by the Soviets and is in their hands, the sources said.

Pelton, arrested and charged with espionage last November, sold the Soviets this secret for $35,000 after he had declared personal bankruptcy and resigned from his low-level NSA job, according to his indictment.

According to intelligence officials, Pelton's betrayal represented one of the gravest American intelligence losses to the Soviet Union. Beyond the operation itself, Pelton's action confirmed one of the intelligence community's worst fears -- that vital high-technology U.S. spy systems can too easily be betrayed by one of the hundreds or thousands of low-level clerks, technicians, translators and others needed to operate them.

When testimony begins in Pelton's trial next week, lawyers on both sides have pledged to keep classified information -- including the identity and description of the program Pelton allegedly betrayed -- out of the proceedings. In a further effort to protect that information, senior administration officials have threatened news organizations with legal prosecution if they reveal details of what Pelton told the Soviet Union.

After several months of extensive reporting, including numerous interviews with present and past government officials, The Washington Post recently prepared a more detailed version of this article to describe what U.S. sources said Pelton had told the Soviets. Several Reagan administration officials, after being asked to comment on that article, strenuously objected to its publication, arguing that it would seriously damage national security.

At a meeting on May 2 with senior editors of The Post, Central Intelligence Director William J. Casey threatened to seek prosecution of the newspaper by the Justice Department if The Post published the story. Casey indicated he would seek prosecution under a 1950 law, never before used against a news organization, that prohibits disclosure of information about communications intelligence -- the interception or encoding of secret communications.

After that meeting with Casey, The Post decided to postpone publication of the article, which had been scheduled for the editions of Sunday, May 4.

On Saturday, May 10, at Casey's request, President Reagan telephoned Katharine Graham, chairman of the board of The Washington Post Co., to urge that The Post not publish the article. Earlier, Vice Adm. John M. Poindexter, Reagan's national security affairs adviser, Lt. Gen. William E. Odom, director of the National Security Agency, and other officials told The Post that publication of the article could endanger national security.

Benjamin C. Bradlee, executive editor of The Post, said yesterday that he continued to believe that the paper's original story would have revealed nothing that was not already known to the Soviet Union. "The capability Pelton disclosed has been mentioned several times in the past in American newspapers," Bradlee said.

But, Bradlee added, because The Post has been unable fully to judge the validity of the national security objections of senior officials, and because of Post lawyers' concerns, the paper has decided to print this article without a description of the technology Pelton allegedly betrayed.

On Monday, Casey asked the Justice Department to consider prosecution of NBC News after it broadcast, on the morning that Pelton's trial began, a brief account of what the network said was the intelligence operation Pelton disclosed to the Soviets.

Pelton, 44, resigned his NSA job in July 1979 after declaring personal bankruptcy. He allegedly contacted the Soviet Embassy in Washington six months later, and disclosed the intelligence operation during meetings with Soviet intelligence officials, according to government affidavits in the case. One source said Pelton had an "extraordinarily sensitive low-level job" within NSA's operations directorate where he worked in the "Soviet group," which consists of 800 to 1,000 employes.

The information collected by the technology Pelton allegedly described to the Soviets was sent to NSA where analysts such as Pelton, who spoke Russian and was trained in computer operations, translated and studied the fruits of the operation.

According to court documents filed in his case, Pelton had been cleared at NSA for top-secret signals-intelligence programs and for other "sensitive compartmented information." A year before leaving the agency, Pelton authored a highly classified technical study of NSA programs relating to the Soviet Union.

An FBI agent's affidavit prepared for the preliminary hearing in Pelton's case stated that "on several occasions from January 1980 through January 1983, Pelton met with Anatoly Slavnov, an intelligence officer of the KGB, and, in exchange for payments described in other government documents as $35,000 in cash , provided to the Soviet agent extremely sensitive classified information related to United States intelligence activities directed at the Soviet Union."

Nearly all the details about Pelton's dealings with the Soviets reported in this article come from FBI accounts of conversations with the alleged spy. Pelton has acknowledged that he has made damaging admissions to FBI agents and has not challenged the substance of the FBI accounts of those conversations. Pelton contends that he was tricked into making the statements by FBI agents who led him to believe they wanted to use him in counterintelligence work.

Pelton was indicted, in part, under a section of U.S. law (18 USC 798) that applies exclusively to the unauthorized disclosure of classified communications intelligence and codes. This is the same law Casey indicated was relevant when he threatened to recommend prosecution of The Post if the newspaper published the article describing the operation Pelton compromised.

Because of the sensitivity of the operation, intelligence officials had what one senior source called a "battle" with the FBI and the Justice Department over whether to prosecute Pelton after the FBI had identified him as the man who betrayed the secret to Moscow. Finally, Justice was allowed to proceed against Pelton, the sources said, because the administration is intent on a widespread crackdown on espionage.

Pelton's alleged espionage might have gone undetected and the compromise of the American intelligence operation might have remained unexplained were it not for the arrival in the United States last August of Soviet defector Vitaly Yurchenko, a colonel from the Moscow headquarters of the KGB. Yurchenko defected to the United States last summer and provided details that FBI officials say led them to Pelton. Yurchenko abruptly decided to return to the Soviet Union in early November last year after three months of debriefing by the CIA. Pelton was arrested days later, on Nov. 24, in an Annapolis hotel.

Yurchenko had been posted at the Soviet Embassy in Washington between 1975 and 1980, where he served as chief security officer. He participated in one of the Soviets' first contacts with Pelton, according to the FBI.

Under the authority of national security wiretap statutes, the FBI eavesdropped on two telephone conversations between Pelton and Soviet Embassy officials. Pelton's voice was identified by FBI voice analysis tests and by the testimony of "several individuals," according to court records.

One of these conversations took place between Pelton and Yurchenko, who was still at the Soviet Embassy in Washington in early 1980, working as the KGB's chief of embassy security. Last fall, after Yurchenko defected, he was given a polygraph examination in an attempt to verify the information he apparently provided about his telephone conversation with someone who had called the embassy. Sources said Yurchenko did not know Pelton's name but FBI officials have said publicly that Yurchenko provided important leads that aided in the identification of Pelton as an alleged Soviet espionage agent.

According to affidavits filed by FBI counterintelligence agents, Pelton made his first contact with the Soviet Embassy on Jan. 14, 1980, about six months after he had resigned from the NSA.

On Jan. 15, 1980, Pelton visited the Soviet Embassy in Washington, the court documents allege. "On that occasion," according to one FBI affidavit, Pelton "agreed to provide sensitive information relating to United States intelligence activities in exchange for cash payments . . . [and] he provided specific information" the same day "relating to a[n] . . . intelligence collection project targeted at the Soviet Union."

During this visit to the Soviet Embassy, FBI agents described Pelton as having a "nervous demeanor." He later admitted that he had disclosed to Soviet officials "a location where the Soviets were losing intelligence information to the United States," the agents alleged.

In the FBI affidavit, the intelligence collection system Pelton compromised during his Soviet Embassy visit was not identified. But the agents stated that Pelton said he "selected the project for disclosure during his initial meeting with the Soviets because it involved a matter which they could readily understand without much technical discussion."

Before leaving the Soviet Embassy, Pelton arranged to travel to Vienna later in the year to provide additional information to Soviet intelligence officials. At the conclusion of the embassy meeting, Soviet officials tried "to secrete the visitor out of the embassy without detection by U.S. authorities," the FBI documents said.

During pretrial hearings on April 17 and 18, an FBI agent elaborated on this statement. The agent testified that Pelton had grown a beard at the time of his initial visit to the Soviet Embassy on 16th Street NW. After his interview with Soviet officials, Pelton shaved off his beard inside the embassy and donned clothes similar to those worn by embassy workers in order to slip undetected out of the building (which is carefully watched by the FBI).

Pelton then boarded an embassy shuttle bus with other workers who are transported back and forth between the downtown embassy and their living quarters on Tunlaw Road NW. Pelton was fed and then returned to downtown Washington where he could return unnoticed to the car he had parked on a street.

Pelton's first alleged espionage trip to Vienna occurred during October 1980. The second was in January 1983 when he was allegedly paid $15,000 for information. On each of these trips, " Pelton spent approximately three to four days in Vienna and was housed at the apartment of the Soviet ambassador to Austria within the Soviet Embassy compound," according to an FBI affidavit. Pelton told the FBI that Soviet KGB officer Anatoly Slavnov was "one of the Soviet officials with whom he had worked in Austria," according to the FBI affidavit.

Each day he was in Vienna, Pelton "spent approximately eight hours a day in debriefing sessions . . . providing written answers to written questions . . . about practically every area of sensitive information to which he had access through his employment at NSA," the affidavit said.

FBI agents showed Pelton the highly classified technical study about NSA programs he had authored in 1978 and asked him how much information from the document he had conveyed to the Soviets. "Pelton stated that the Soviet agent was interested in all of the programs outlined in that document," according to the affidavit.

Pelton made a final trip to Vienna in April 1985 to meet with his Soviet contacts, but during his three-day stay in Austria he somehow missed connections with KGB officials. Last July, Pelton received an overseas telephone call "relating to his failure to meet with the Soviet agent in April of 1985 in Vienna and requesting that he make another trip to Vienna in October 1985," according to the FBI documents.

But Pelton did not make the trip. One source indicated that Pelton did not go because he felt he had told the Soviets everything he knew and had been thoroughly debriefed.

Before Pelton was arrested Nov. 24, he told the agents that he had acted on impulse in approaching the Soviets. The agents said Pelton acknowledged that his disclosures were costly and harmful to the United States, and he added, "they got more out of me than I wanted to give up," according to the affidavit.

U.S. intelligence officials have not been able to interview Pelton since those interrogation sessions in November. Pelton's court-appointed attorneys have advised him to exercise his rights to have no further discussions with the FBI or federal prosecutors, government sources said.

At the pretrial hearing last month, Pelton asserted in testimony that his statements to FBI agents were not voluntary and that his constitutional right against self-incrimination had been violated.

Pelton, who grew up in Benton Harbor, Mich., spent four years in the Air Force in the early 1960s. He learned Russian at the Air Force language school in Bloomington, Ind., and later became a cryptologic technician, joining the NSA in 1965. After leaving the agency in mid-1979, he worked as a yacht salesman in Annapolis and later as a computer consultant. He also tried to set up his own international finance company.

Pelton and his wife separated last August. They have three grown daughters and a teen-age son. Pelton has testified that he developed a dependency on alcohol and Dilaudid, a heroin-like narcotic, in 1984 and 1985, when he was dating a woman with drug and alcohol problems.

In recent years there have been three similar cases in which low-level figures sold extremely sensitive secrets to the Soviets. In 1977, Christopher J. Boyce, a 23-year-old clerk in a top-secret vault of TRW Inc., the defense contractor, was arrested for selling a Soviet agent the plans for the Rhyolite communications intercept satellite, one of America's most expensive and sensitive pieces of espionage hardware.

Boyce's compromise was compounded when the British government discovered in 1982 that a low-level Russian-language specialist once employed by British intelligence, Geoffrey A. Prime, had sold the Soviets copies of messages intercepted by Rhyolite satellites for six years, giving the Soviets an extensive understanding of U.S. satellites' capabilities.

In 1977, William J. Kampiles, then 23, was arrested for selling the technical manual for the KH11 photoreconnaissance satellite, the most advanced U.S. model, to the Soviets for $3,000. Kampiles had worked for eight months as a watch officer at the CIA.