The State Department yesterday retaliated against the Soviet Union's expulsion of an Associated Press correspondent from Moscow by ordering the ouster of a Tass correspondent from Washington.

Press spokesman Frederick Z. Brown said the action was taken with regret, and called the course of events "a step backward from the objective of improving working conditions for journalists contained in the Helskinki [agreement] final act, and from the more fundamental interest of promoting the freer flow of information."

On Friday, Associated Press correspondent George Krimsky was ordered to leave Russia within a week by the Soviet Foreign Ministry. The Soviets accused Krimsky of espionage activites and currently violations, but the AP, denying the charges, said they were a pretext for getting rid of an aggressive Russian-speaking reporter.

Secretary to State Cyrus R. Vance protested to Soviet Ambassador Anatolly F. Dobrynin Friday after receiving word of Krimsky's ouster, the first expulsion of an American correspondent from the Soviet Union since 1970.

Yesterday morning Senate Majority Leader Robert C. Byrd (D-W. Va.) told reporters, "We ought to tell one of their people right away to start packing his bags. I believe in retaliation to this sort of thing."

Shortly after noon the State Department notified the Soviet embassy that Vladimir I. Alekseyev, a member of the 18-member Washington bureau of the Soviet news agency, Tass, was being ordered to leave the country within a week. The State Department specified that the action was being taken in response to Krimsky's ouster.

Asked if Tass will be able to replace Alekseyev with another correspondent, Brown said he assumes that they will if the Associated Press is also allowed to replace Krimsky in Moscow.

Soviet colleagues said Alekseyev, 34, came here last April with his wife and daughter. He has been covering congressional and State Department news.

Rep. Robert F. Drinan (D-Mass.) called the ouster of Alekseyev "very unwise," and said that the U.S. action violates the Helsinki accords, which say that jurnalists in "legitimate pursuit of their professional activities" should not be subject to expulsion.

Five U.S. correspondents were expelled from Moscow in 1968-70, and three Soviet correspondents were expelled from the United States in the same period.

The decision to retaliate for the ouster of Krimsky was taken by the State Department after consulation with the White House, sources said. The Associated Press reported that President Carter had approved the action.

In a related development, the State Department had no immediate comment on the arrest in Moscow Friday of Soviet poet Alexander Ginzburg. However, exiled writer Alexander Solzhenitsyn, in a statement from his home at Cavendish, Vt., charged that the arrest "reflects the decision of the Soviet authorities to crush by hunger and poverty hundreds of families of persecuted and imprisoned people and to force thousands of others into fear and silence."

Ginzburg was the main representative in the Soviet Union of the Russian Social Fund, organized by Solzenitsyn before his exile, to aid political prisoners and their families. It draws on the royalties of Sozenitsyn's books abroad.