The Reagan administration, dramatically raising the stakes in its clash with Moscow over diplomatic expulsions, yesterday ordered 55 Soviet diplomats -- the largest number ever expelled from the United States -- to leave the country by Nov. 1.

Five of the Soviets were declared persona non grata in direct response to the expulsion Sunday of five American diplomats from the Soviet Union. In addition, the United States ordered 50 more Soviets out of the United States to reduce the personnel at the Soviet Embassy here and the Soviet consulate in San Francisco to the same number of Americans stationed at the U.S. Embassy in Moscow and consulate in Leningrad.

Expulsion of the 55 marked the third round in an escalating contest that began with a U.S. campaign to cut back the size of the Soviets' U.N. Mission on the grounds that it is a haven for spies. Using the diplomatic expression for espionage, State Department spokesman Charles E. Redman said the 55 were engaged "in activities incompatible with their diplomatic status."

However, while Redman said the Soviets had been warned that the United States would take such action if Moscow retaliated for the expulsion of 25 members of its U.N. Mission, administration officials said the State Department had wanted a less dramatic response than the one taken yesterday to minimize chances of undermining superpower relations.

In a brief initial reaction, the Soviet news agency Tass condemned the latest expulsions as "another step aimed at worsening Soviet-American relations." Earlier yesterday, before the U.S. action was announced, Soviet Foreign Ministry spokesman Gennadi Gerasimov said in Moscow "that if the United States insists on a continuation of the game according to the tit-for-tat principle, all this may go on to infinity." [Story on Page A31.]

President Reagan, at a White House dinner last night honoring West German Chancellor Helmut Kohl, said of the ousted Soviet envoys, "We feel that they were connected with intelligence operations, the KGB and so forth. Besides, they had more than we did and we just leveled it off."

Administration sources said the expulsion decision was a victory for CIA Director William J. Casey and Attorney General Edwin Meese III. Both had urged the White House on Monday to make a strong response to the Soviet expulsion of five Americans.

The intelligence and law-enforcement communities earlier this year began a concerted effort to curb Soviet espionage activities in the United States. This effort resulted, in part, in the arrest of Soviet spy Gennadi Zakharov in New York last August, which triggered the Soviet detention of American reporter Nicholas Daniloff.

The intelligence and law-enforcement agencies wanted to try Zakharov to make a point to the Soviets and to the American people about curbing espionage, the sources said, but the administration decided to allow Zakharov to plead no contest to espionage charges as part of a swap for Daniloff. Yesterday's announcement, one official said, came about because the White House promised Meese and Casey "the next time this happened, we'd do it right."

A Justice Department official said Meese was a "strong advocate of the steps that were finally taken." This official said the State Department was "foot-dragging" but "it got turned around in the meeting at the White House."

At the White House meeting Monday, Reagan met with Secretary of State George P. Shultz, Vice President Bush, Meese, Casey, and White House chief of staff Donald T. Regan.

The administration took the position in public remarks that its move should not affect the larger U.S.-Soviet relationship. Reagan, in welcoming Kohl yesterday, said there was "ample reason for optimism" that Washington and Moscow will reach a nuclear arms reduction agreement.

The U.S. action came 10 days after the collapse of the Iceland summit between Reagan and Soviet leader Mikhail Gorbachev set back hopes of improving U.S.-Soviet relations. It thus raised immediate questions about whether the Soviets will feel compelled to respond in ways that could further worsen superpower relations and make it more difficult to negotiate arms control agreements.

In the aftermath of the Iceland meeting, Gorbachev had appeared to be making a number of conciliatory gestures toward Washington. He signaled possible greater flexibility on arms control talks and permitted the departure from the Soviet Union of a prominent Jewish dissident and a Jewish woman seeking to donate bone marrow to a critically ill brother in Israel.

Even the order Sunday expelling the five Americans was regarded in diplomatic circles as a minimal response since Moscow had announced publicly that it would retaliate if the United States went through with the expulsion of the 25 Soviets at the United Nations. Gerasimov, who concluded his remarks by saying "it is best to draw a line under this affair," appeared to be signaling Moscow's desire for the United States to refrain from the kind of action that would require a further Soviet response.

Redman said yesterday that despite the latest expulsions the United States "remains committed to pursuing the dialogue from the Reykjavik meeting in all areas of our relationship."

But, he added, "at the same time, as you can see by this action, we will certainly do everything we can to prevent this country from being used as a haven for espionage."

Specifically, he said, the U.S. action will shrink the previous authorization that permitted the Soviets to keep 320 diplomats in this country, exclusive of their U.N. missions, to a new lower total of 251 -- the same number that the United States has in the Soviet Union. The Soviets now will be allowed a staff of 225 at their embassy here and 26 at their San Francisco consulate.

Although the United States specifically told the Soviets which diplomats must leave, Redman publicly identified only the five who were declared persona non grata in response to the expulsion of the five Americans. He gave their names as Vasily Fedotov, Oleg Kikhachev and Aleksandr Metelkin, all counselors at the embassy here; Nikolay Kokovin, an attache at the embassy, and Lev Zaytsev, a consul in San Francisco.

A White House official said that last month, when the United States and the Soviet Union were arguing about the expulsion of the 25 Soviets at the United Nations, the Soviets were told explicitly that Washington considered the Soviet presence at the United Nations as separate from diplomats directly accredited to the governments. As a result, the official said, the Soviets were warned that if they retaliated against American diplomats in Moscow, the administration would apply the parity principle called for in legislation passed by Congress last year.

Redman also said that "the Soviets were aware at the time of that action at the U.N., that if they took action in retaliation that we would then be moving to establish parity at the levels I have described." But, while he repeated several times that the Soviets had been warned, he refused to say directly whether the United States had intended to invoke a policy of full parity or a partial movement in that direction.

Other State Department officials, while affirming their understanding that the Soviets had been given a specific warning about the consequences of expelling any Americans, said that the department had advocated less stringent retaliatory measures.

They said that while the department supports the idea of parity in the U.S. and Soviet diplomatic representation, senior State Department officials believed that this goal should have been implemented in a phased manner in the interest of ending the round of expulsions before it got out of control.

In addition, these officials said, the department is in the process of trying to improve security at the embassy in Moscow by replacing Soviet employes who perform clerical and maintenance work with nondiplomat Americans. The officials said the department expected it would take about 1 1/2 years to implement that policy in a significant way and is fearful that the round of expulsions will cause major setbacks in the program.

A White House official, familiar with the Monday meeting where the expulsion order was decided on, said it was "decided that the only way to do business was to follow through . . . to be true to our word." A State Department official added that "some tradeoffs were made that enabled us to join in the decision," but he declined to say what the tradeoffs were.

Staff writers David Hoffman, Howard Kurtz and Donnie Radcliffe contributed to this report.