FBI agents who keep track of about 1,300 intelligence operatives from Communist countries in the United States are overworked and sometimes overwhelmed in their efforts to prevent espionage, according to intelligence experts and members of Congress specializing in intelligence.

Times have changed since the 1950s, when there were only a few hundred Soviet bloc personnel in the United States, and the FBI had approximately as many counterintelligence agents on the street as there were known agents of the KGB, the Soviet intelligence agency. Nowadays, according to a spokesman for the Senate Intelligence Committee, the ratio of our agents to the number of people they have to watch is "unquestionably unfavorable" for the United States.

"From the Senate Intelligence Committee's point of view, there's no question we've not recovered sufficiently from the time a few years ago when we cut back on our intelligence efforts, at the same time the Soviets were increasing their numbers here," a committee staff member said.

Even a one-to-one ratio, designed to ensure ability to follow each hostile agent, "doesn't begin to answer the problem," said Rep. Lee Hamilton (D-Ind.), the House Intelligence Committee's chairman. "The problem is monumental . . . . We have some catching up to do."

For their part, FBI officials say they are handling the workload.

"We can use more people and resources, but we're certainly not overwhelmed," said Edward J. O'Malley, chief of the FBI's intelligence division. "We're not daunted by it."

The exact numbers of FBI counterintelligence agents is classified, and O'Malley last week described as "grossly understated" published reports that there are only 300 to 400 such agents. The FBI has substantially beefed up its budget for counterintelligence agents in the past five years.

The Walker spy case, in which retired Navy warrant officer John A. Walker Jr. and three associates are charged with espionage, has prompted a new round of concern on Capitol Hill, where the Senate and House Intelligence Committees have been holding closed-door hearings for several weeks on the United States' spy-catching effort.

"The Walker case illustrates the FBI's difficulties," said Sen. Patrick Leahy (D-Vt.), the Senate Intelligence Committee's vice chairman. "The number of Soviet and East European officials in this country is simply too great for the FBI to monitor their activities adequately." An Unglamorous Job

Foreign counterintelligence, or "f.c.i.," which seeks to locate spies and neutralize or manipulate them, and sometimes arrest them, is a complex and mistrustful line of work, replete with false defectors, double agents and other trickery.

But it is far from the glamorous portrait of spies and counterspies painted by many movies and novels.

"The business of counterespionage is a Dantean hell with 99 circles, and the men who dare its enigmas without exception have thick glasses, a midnight pallor, stomach ulcers, a love of fly fishing, and fretful wives," wrote journalist Thomas Powers in a book about the CIA.

Most of the work of FBI counterintelligence agents is quite humdrum, involving almost 24-hour-a-day monitoring of agents of the KGB and intelligence agencies of other communist countries. Using high-technology gear, FBI agents tap their telephones, listen in to their offices and cars, intercept their coded messages, photograph visitors to their embassies, and "tail" the intelligence officers in cars wherever the agents go.

"The basic idea of counterintelligence is to make it riskier and more time-consuming for the other service to operate," said one retired ranking FBI counterintelligence official. "You get awfully good at crossword puzzles and eating in your car."

There are the constant decisions whether to assign agents to watch, for example, the Polish diplomatic couple driving across the country for a vacation or the suspicious Soviet scholar at the trade conference, FBI officials said.

"One continuing fact of counterintelligence is you're almost always behind," one said. "You're always in a sense pushing to narrow that gap."

Starting with the era of peaceful coexistence in the 1960s, the number of communist nations' officials in this country, including embassy and consular employes, United Nations representatives, trade officers and journalists, has shot up. It has doubled to almost 4,000 in the past decade, the FBI's O'Malley said. He and other intelligence experts estimate that one-third of them are trained intelligence agents.

Most FBI surveillance cars have at least two agents each. More sensitive cases, in which the one followed may think he or she has shaken the pursuers, can involve 50 or more agents in cars.

Even at best, the FBI is able to track the movements of only known Soviet bloc agents. Many other suspected ones operate under less scrutiny.

These include hundreds of non-KGB embassy personnel and members of foreign officials' families, tens of thousands of Soviet bloc emigres who live in the United States, and tourists, scholars, artists and sailors who pass through.

Soviet officials warn many of them before they leave the Soviet Union that they must help the KGB if asked.

"Anyone who refuses to work for the KGB risks not getting a passport," said Roy Godson, a Georgetown University political science professor, intelligence expert and government consultant.

Such non-KGB "co-optees" commonly are used to research topics for the KGB, to scout recruits or to set up "dead drops," locations where agents exchange messages with contacts, officials said.

A special source of concern are the approximately 9,500 emigres from communist countries who work in defense industries and have security clearances, officials said. Some with family members in the old country could be vulnerable to coercion from foreign agents, they said.

Another category is the so-called "illegals," Soviets who covertly enter the United States and pose as normal citizens, often in hopes of gaining access to secrets over a lifetime. One "illegal" was Rudolf Herrmann, a KGB colonel who spent 18 years posing as a free-lance photographer in Canada and the United States before being turned into a U.S. double agent and then going public with his story in 1980.

At the heart of counterintelligence in the United States is the task of tailing the known KGB agents, most of them based in Washington, New York and San Francisco.

Soviet Embassy officials here are free to travel without restriction within a 25-mile radius of Washington, but they must notify the State Department about travel in most other parts of the country. Eastern European officials here, as well as United Nations employes from Soviet bloc nations, can travel anywhere without notification. Counterintelligence officials say this is one reason the United Nations has become what President Reagan called a Soviet "spy nest," and some want this loophole closed.

In Washington, many of the car trips start at the Soviet Embassy on 16th Street NW, or the new Soviet compound on Wisconsin Avenue. FBI agents watch the comings and goings at these locations from apartments or offices nearby, sources said.

The FBI's hope in this cat-and-mouse game, which is played out with roles reversed in Moscow and Leningrad, is that FBI agents here will follow a Soviet agent to a "dead drop," or even a meeting with a contact.

"They spread out their meetings over a long period of time," said William Branigan, a retired ranking FBI counterintelligence official. "If they meet maybe once a year, chances are one in 365 of catching them."

FBI agents file detailed reports on the habits of their KGB prey, where they go, with whom they talk. The two sometimes gain a kind of regard for one another. Sometimes if a KGB agent gets lost, the FBI tail might flag down the agent and give directions, several veterans said.

KGB agents often spend hours -- sometimes half a day -- driving, turning, stopping, then taking off again in an attempt to lose their tails, or to "dry-clean" themselves.

And some of the FBI's best spy catches have come from very tedious electronic eavesdropping, something officials say little about. For example, authorities discovered that an Air Force lieutenant, Christopher Cooke, was delivering nuclear secrets to the Soviets after he used a phone in the Soviet Embassy in 1980 to tell his father he was having car trouble. The FBI was listening. Standards Questioned

The FBI's present counterintelligence operations point to the case of Richard W. Miller as an example of problems.

Miller was a burned-out FBI agent in Riverside, Calif., with a horrendous job rating in 1982 when his superiors came up with a new job for him -- counterintelligence. Transferred to Los Angeles, he was supposed to help watch the local emigre community.

Last year Miller became the first FBI agent ever charged with espionage. He fell in love with a Soviet emigre housewife, Svetlana Ogorodnikova, and is charged with giving her an FBI manual to be passed to the Soviets. She and her husband have pleaded guilty, and Miller is awaiting trial.

"He was a flop, and look what they do, they put him in counterintelligence," said Francis McNamara, a conservative national security expert and author of a recent book critical of U.S. counterintelligence. "It's an indication that the FBI doesn't take it very seriously."

While other intelligence specialists think the FBI does care about counterintelligence, they agree that the Miller case is a sign its quality has slid in the bureau.

It's never been a glamorous field. Most of its FBI practitioners work in high-cost areas, and the FBI gives no pay differential according to location. In addition, especially in the age of J. Edgar Hoover, FBI promotion often depended on numbers, such as bank robbery arrests. The counterintelligence unit, which might make only two arrests a year, sometimes seemed like a backwater, retired agents said.

But the real trouble started when the FBI and CIA began performing unauthorized or illegal investigations of U.S. citizens in the 1960s, experts said. The public became suspicious about everything the CIA and FBI did.

The FBI and CIA's counterintelligence divisions became "deeply suspect" in the public eye, and "the net effect for agents is it could not have been a more powerful disincentive to get the hell out," said Robert T. Crowley, a retired CIA official and writer on intelligence matters.

Starting with attorney general Edward Levi's guidelines a decade ago, there was a wave of new rules and executive orders restricting the FBI, especially in the area of investigating political groups. Essentially, the guidelines, which have been loosened somewhat but not drastically by the Reagan administration, have said authorities must have evidence of a crime before they can begin investigating such organizations. The foreign counterintelligence guidelines are classified, but it is agreed that they allow more leeway.

There is disagreement in the intelligence community about whether guidelines restricting investigation of political groups, such as a Communist organization or one that promotes the ideology of foreign powers, hurts the nation's ability to ward off spies. Top FBI officials say they do not, but some conservatives and retired intelligence officials disagree.

Ray Wannall, a retired top FBI counterintelligence official, said that the FBI in 1973 had 21,414 domestic security cases under investigation, while last year, because of the guidelines, it had 25.

Wannall called the guidelines a "Catch-22 . . . . There's no way to know whether a group is engaged in activities inimical to our government without conducting an investigation -- which is precluded under the guidelines."

"Because so many subversive groups are not now under investigation," Herbert Romerstein, an intelligence expert and former House Intelligence Committee staff member wrote in a 1980 essay, "the domestic intelligence data base has atrophied alarmingly since 1976."

The criticism of the FBI in the mid-1970s, as well as the indictment of some FBI officials for unauthorized investigative techniques and laws making agents personally liable in the event of abuses, demoralized many in the FBI counterintelligence staff, specialists and former officials said.

Budget cutbacks, plus mandatory retirement at age 55, caused large staff reductions in FBI counterintelligence in the 1970s, and many of those who left were the most experienced agents, experts said.

"They took with them the institutional memory," Wannall said.

"As a result of wholesale retirements and inadequate emphasis on counterintelligence in the 1970s, relatively few veterans remain, and the experience level has dropped to an appallingly low level," wrote intelligence expert John Barron in a 1983 book, "KGB Today." "On any given day, the Soviet bloc can send into the streets of Washington and New York more professional intelligence officers than the FBI can deploy against them."

FBI officials said that despite problems in the past, increased resources for counterintelligence in the past several years have bolstered morale, added staff, upgraded high-technology intelligence-gathering tactics and improved training.

"Our people today are the best-trained counterintelligence people in the world," O'Malley said. "Agents throughout the field are asking to work in this business."

But some members of Congress think a lot more needs to be done.

Leahy said that, among other things, the FBI must make counterintelligence as desirable a career path for FBI agents as working on organized crime, for example, and must pay agents in high-cost areas more money.

"We have to say people's careers will not be hurt by going into counterintelligence," Leahy said. "We need cases where people in CIA and FBI get promoted because they're good at counterintelligence."

Leahy last week helped engineer an amendment to the State Department's authorization bill, now awaiting Reagan's signature, that would help the United States' counterintelligence problem. The idea is to equalize the number of Soviet diplomats here with the U.S. diplomats in the Soviet Union.

There are 279 Soviet diplomats in their embassy here, and only 184 Americans in the embassy in Moscow. In addition, the U.S. Embassy there employs 214 Soviets, most of them presumed to be working for Soviet intelligence. Few Americans work in the Soviet Embassy here.

One intelligence specialist said that the United States is outclassed because the Soviet Union spends much more money on its spies than the United States does on counterintelligence. "We're clever and energetic, and we have some stars," he said. "But we're like a college basketball team playing against the NBA. It's a different kind of game."

O'Malley, who is retiring this month, said that with increased resources, the FBI has been able "to make it a lot more difficult for a person to commit espionage. But no one can say this is the end of espionage. Anyone who says that is a fool."