The State Department and Justice Department are locked in a sharp, behind-the-scenes dispute about whether the Carter administration's moves against Soviet spies in the United States are fueling the deterioration of U.S.-Soviet relations.

At issue is a problem that has plagued U.S. governments since the height of the Cold War - the clash between the pursuit of better relations with the Soviet Union on the international scene and the need to safeguard national security.

In recent weeks, the conflict has been exacerbated by what high-ranking State Department officials contend is an increasingly hard-line approach in Attorney General Griffin B. Bell's Justice Department toward espionage cases.

State officials say that some recent actions by the Justice Department are regarded in intelligence circles as a changing of the "unwritten rules" by which the superpowers traditionally have played the spying game.

In addition, the State officials charge that the FBI, which is under Bell's control, has been trying, with the help of sympathizers in Congress, to pull off what one State source calls a "power grab." Its aim, State officials say, is to give the FBI the dominant voice in deciding whether foreign nationals suspected of espionage activity should be denied visas to enter the United States.

On the other side, Justice and FBI officials deny that there has been a major shift in policy toward Soviet espionage cases. They do concede, though, that their responsibility for safeguarding national security and enforcing the law frequently runs counter to what one calls "the convenience of the State Department."

In diplomatic and intelligence circles, this gap between the attitudes of State and Justice is regarded as one of the major underlying causes of the recent sharp increase in tensions between the Carter administration and the Kremlin.

Its effects were made vividly clear by the events that began unfolding in June when the FBI, amid a blaze of front-page publicity, arrested two Soviet citizens employed by the United Nations and charged them with trying to buy secret U.S. naval warfare documents.

The decision to prosecute the two on espionage charges was preceded by a lively debate within the administration that pitted State and Central Intelligence Agency against Justice and the FBI. The debate ultimately had to be resolved by President Carter.

The State Department, backed by the CIA, argued that the two Russians should have been quietly expelled rather than arrested and prosecuted. Department sources say Carter was warned that prosecution would be interpreted by the Soviets as a breach of the "unwritten rules" that the two countries normally apply to each other's apprehended agents and would invite a dramatic and inflammatory Soviet retaliation.

"As recently as two years ago, when [then-Secretary of State] Henry Kissinger was calling the shots on foreign policy, that would have been the paramount consideration," says one State official. "It would have been handled quietly in a way designed not to disturb the larger framework of our pursuit of detente with the Soviets."

However, in making a decision, Carter sided not with State but with Bell, who reportedly went to the president to argue that the case involved a clear violation of law that could not be overlooked. Bell also is known to have argued that the so-called "rules" didn't apply since the two Russians did not have diplomatic status or immunity.

The retaliation feared by State came in July when Soviet police, in an act that sent U.S.-Soviet relations plunging to their lowest ebb in years, dragged an American businessman, Francis J. Crawford, from his car on a Moscow street and charged him with currency violations.

Crawford's arrest forced Washington into a frenzy of high-level negotiation with the Soviets that finally resulted in Crawford and the two Russians being released temporarily in the custody of their ambassadors.

"What was accomplished by arresting the Russians?" asks the State Department official. "All it did was trigger an incident that unnecessarily tied up much of the U.S. government including the president and the secretary of state, in a wasteful and enervating effort to try and undo the damage."

"In addition," he notes, "the American business community in Moscow has been paralyzed with fear and uncertainty that could seriously set back the development of U.S.-Soviet trade; suspicions and animosities have been heightened on both sides, and, in an atmosphere like that, the chances for reaching accommodations on matters affecting world peace aren't enhanced."

he points out, by way of illustration, that the two Soviets are scheduled to go on trial Sept. 12 before a federal court in New Jersey. On Sept. 19 - a time when the trial is likely to be in full swing and attracting a heavy share of press attention - Secretary of State Cyrus R. Vance and Soviet Foreign Minister Andrei Gromyko will be meeting across the Hudson River in New York to continue their search for a strategic arms limitation talks (SALT) agreement.

"The fallout from the trial will be swirling all around them," the official predicts. "How can you make progress in a situation like that?"

Some State and CIA officials express hope that the incident might have been a "learning experience" that will cause Carter and Bell to take a "less simplistic view" toward espionage cases in the future. However, the Justice Department gives no outward sign that it's had any second thoughts on the matter.

Instead, one Justice official who is privy to Bell's thinking, says: "We never thought there was much room for argument on this one. In this administration, when someone's caught in a serious violation of law, he's going to be prosecuted - no matter who he is or where he comes from. That's how Bell feels, and he belives the president feels the same way."

The cleavage between the two departments is also likely to affect efforts to resolve another aspect of the controversy - that involving U.S. visa policy and charges that the State Department's permissiveness has allowed too many spies from the Soviet Union and its communist bloc allies to infiltrate the United States.

In recent weeks, these charges have triggered a number of moves by conservatives in Congress to restrict and inhibit State's visa-granting powers by giving more weight to the FBI's contention that potential security risks should be kept out.

The House, for example, has passed legislation that would make the House and Senate Intelligence committees custodians of a list supplied by attorney general and containing the names of all aliens admitted to the country over the objections of the FBI. Legislation for a similar but much less detailed list also has been adopted by the Senate.

Last month, the Senate adopted an amendment offered by Minority Leaser Howard H. Baker Jr. (R-Tenn.) to the military assistance bill that would have repealed earlier legislation relaxing restrictions on visas. However, the Baker amendment later was dropped by a House-Senate conference committee.

Then, on July 28, the Senate Appropriations Committee, in a report to the full Senate, included language that one Justice Department officials describes as "a kick in the groin for the State Department."

In its section on funds for the Immigration and Naturalization Service, another agency under Justice Department control, the committee said it had "determined that far too many Soviet block intelligence personnel have been admitted entry into the United States."

"Therefore," the report continued, "the committee directs that the Immigration and Naturalization Service under the guidance, control and supervision of the attorney general deny entry and enforce expulsion of hostile intelligence service personnel irrespective of visas issued by, or policies of, the Department of State."

These various moves to give federal law enforcement agencies the dominant say over visa policy are viewed by the State Department as an even greater threat to its ability to deal with the Soviets than the conflict over espionage prosecutions. The department's ability to grant visas to visitors coming from the Soviet bloc and as delegation members, journalists, students, businessmen or tourists is an important element in its everyday dealings with Moscow and its allies.

In cases where a visa is issued to someone who might be considered a security risk. State is supposed to consult with the FBI to see if the bureau objects to the persons's entry. If State and the FBI then are unable to agree, the matter is referred to INS for a final decision; and, during recent years. State has prevailed in the overwhelming number of disputed cases.

For that reason, some State Department officials say they believe the FBI has played a behind-the-scenes lobbying and orchestrating role in the recent congressional attempts to rein in State's visa-granting powers.

"The FBI has lost hundreds of disputed cases, and they say we never listen to their recommendations on keeping someone out," says one department official. "Now we have pretty good indications that they've been working on friendly congressmen to get the rules changed to give them the upper hand."

"The problem," he charges, "is that the bureau wants to keep out everyone who ever walked down a street where there was a communist in the vicinity. More than 90 percent of the objections they make are based on the flimsiest evidence - on memberships or associations that were ended years ago or on unprovable second or third-hand allegations made by unreliable sources."

"To follow such a policy would be totally counterproductive to our foreign policy goals," the State official argues. "It invites retaliation against Americans traveling to communist countries, and it gives the Soviets propaganda to counter our human rights policy. Every time Congress takes a regressive step on visas, the Soviets can respond by asking, 'Who are you to criticize us for our alleged restructions on freedom to travel?'"

On the other side, FBI and Justice Department officials deny they're engaged in a campaign to tilt visa policy toward national security considerations. They admit, though, that FBI Director William H. Webster has been "totally candid" when asked for the bureau's opinion by various committees of Congress.

They say Congress has put pressure on the FBI to step up its combating of foreign intelligence and that, in turn, has severely taxed the bureau's resources because of the need to keep some kind of check on the hundreds of alleged potential security risks the State Department has allowed in the country.

"Webster went to Congress not long ago and asked for 125 new agents to keep up with the job," an FBI official points out. "They hit the ciling and asked him, 'Why do you need more men? Why don't you keep them out in the first place so you don't have to watch them?'"

Lawyers from State and Justice recently have been trying to search out a common ground that will permit more harmonious resolution of visa dispute. However, the effort is still in an early stage. As one of the participants says:

"We've agreed that we have to reach an agreement. But there's still a very big gulf to be bridged; and, at this point, it's hard to say if we can find a modus vivendi acceptable to both sides."