The State Department acknowledged yesterday that the Soviet Union and the United States have secretly handled a "great many" espionage cases of the sort that are now being publicized by both sides.

The acknowledgement bolstered what government sources have said privately for the past week: that the "unwritten rules" of spying, which dictated mutual disdretion, are breaking down.

The chance in approach has accompanied a deterioration in Soviet-American relations over the past year. Government experts said they expect further splashes of publicity about espionage cases in the coming weeks.

The latest of these came Monday, when the Soviets revealed that nearly a year ago they detained and then expelled Martha D. Peterson, a CIA agent posing as a diplomat in Moscow, after they caught her allegedly planting espionage devices on a bridge crossing the Moscow River.

The U.S. ambassodor in Moscow reportedly asked the Soviets at the time a avoid publicity about the incident, which they did until this week.

Sources have speculated that the Soviets broke their silence because the United States made public two Soviet spying incidents, including the arrest last month of two Soviet U.N. employes who had allegedly paid $20,000 for secret American antisubmarine-warfare documents.

That arrest, said one knowledgeable source, probably would not have occurred two years ago, when Soviet-American relations were more calm.

"It clearly changed the rules of the game," the source said, and going ahead with it "was clearly a major and serious decision made at the higest levels."

State Department spokesman Tom Reston declined to comment on whether there has been a conscious change of policy in handling such cases. Responding to a question about the Martha peterson case, however, he said that "over the years, many of these cases have not become public." Later he added that "a great many of them just do not crop up."

Government statistics show a signifuant increase in U.S. prosecutions of espionage activity over the past three years. Ten defendants have been prosecuted since June 1975, according to the justice Department, more than during the all of the previous 10 years.

In recent months, the government has moved publicly against the two U.N. employes, a third Soviet diplomat stationed at the United Nations and an alleged Washington-based spying ring designed to funnel information to Vietnam.

Recent newspaper stories revealed the existence of a secret Soviet-operated electronic listening post the south wing of U.S. embassy in Moscow.

All this activity is in sharp contrast to the way things were generally done during the 1960s and much of the 1970s. With a minimum of publicity, if any, the Soviets and Americans would city "circumtances" or "inappropriate behavior" on the part of the other side and declare the offender persona non grata.

A lively debate within the administration preceded President Carter's decision to approve prosecution of the most recent case of alleged Soviet espionage - the arrests of the U.N. employes - according to sources.

State Department and CIA officials argued against arresting the two Soviet officials because they feared retaliation by the Soviet Union.

But Attorney General Griffin B. Bell took the position that the case should be tried because it was a clear violation of law by Soviets not covered by diplomatic immunity, Justice Department sources said.

Through Carter finally sided with Bell, Justice officials said they did not view the decision as a major shift in U.S. policy toward Soviet espionage cases. "It's all being handled on a case-by-case basis," one Justice official said.