Something startling, unusual and probably very important happened in Moscow yesterday, but outside the small group of Soviet leaders no one here seemed to know just what it was.

The superficial facts are known: One of the most powerful men in the Soviet Union, Marshal Nikolai Ogarkov, was suddenly, unexpectedly replaced as chief of staff of the Soviet armed forces and first deputy minister of defense.

But why?

One experienced western diplomat here suggested that Ogarkov's ouster might be the visible tip of some broader upheaval in the Soviet leadership. At the other extreme, different analysts suggested that Ogarkov might simply be ill, or might have been involved in some kind of scandal.

But the circumstances of his removal suggest a sudden decision, which would seem less likely if illness or scandal were the cause. On Wednesday Ogarkov was going about his duties as usual. Thursday evening's edition of Izvestia, the government newspaper, which appears around 7 p.m., reported that Ogarkov had been at a Moscow airport on Wednesday to see off a visiting Finnish general.

Barely two hours after Izvestia was on the streets, the news agency Tass announced that Ogarkov had been removed from his post and replaced by Marshal Sergei Akhromeyev, his chief deputy and apparently his intimate colleague. This sequence confirms that Ogarkov's ouster was sudden, and probably a surprise to Ogarkov himself.

But why did it happen? Moscow was rife with rumors and theories today, none of them confirmable.

Some analysts thought a fight may have erupted between civilian politicians and the military. Some thought the Soviet leader, Konstantin Chernenko, was getting revenge for Ogarkov's support of Yuri Andropov in November 1982, when Andropov beat out Chernenko in a struggle for power after the death of Leonid Brezhnev. (Chernenko became leader after Andropov died last February.) Some thought that Ogarkov's ouster, coming a year and five days after the downing of a Korean civilian airliner by Soviet fighter pilots, was deferred punishment for that episode. Others speculated that the Soviet Army's lackluster performance in Afghanistan may have been a cause of Ogarkov's dismissal.

Another theory is that Ogarkov got too big for his marshal's uniform, prompting the Communist Party leadership to remind him -- and thereby other military men with oversized ambitions -- just who runs this country. This theory is appealing, given Ogarkov's unusually self-confident and forceful manner as well as some of his public statements that suggested he was impatient with civilian leaders who slighted the armed forces' needs.

This theory is certainly plausible, but it was the civilian leadership that gave Ogarkov numerous opportunities to establish himself as an unusually prominent personality here. The civilians chose Ogarkov to speak for the government at a press conference shortly after the Korean jetliner was downed, an assignment he handled masterfully. Later they used him again at another press conference on arms control issues.

In modern times, only one other official here has been permitted to speak in the name of the Soviet government before live television cameras in press conferences on important issues -- Foreign Minister Andrei Gromyko. It would seem odd for the same leadership that gave Ogarkov such authority to suddenly decide he had become too influential.

Moreover, senior officials here acknowledge surprisingly freely that the military has indeed emerged in a more powerful role in recent times. One senior official observed recently that "in times of tension, it is natural for the military to be more visible." However, this comment does not preclude the possibility that the civilian leaders had become anxious over the appearance of ascendant military influence in the country's affairs.

Ogarkov's ouster is one of several unusual signs recently that suggest something is amiss in the Kremlin. It is clear, for one, that Chernenko is in poor physical condition and is unable to perform his duties forcefully. In an appearance on television Wednesday night, his first in more than seven weeks, Chernenko looked like a frail and nervous old man who was barely up to awarding medals to three cosmonauts, let alone providing dynamic leadership to this huge country.

But if Chernenko is weak, some diplomats noted with interest that the decision to oust Ogarkov was apparently made Thursday at the first Politburo meeting since Chernenko returned from a vacation and a brief hospitalization for heart trouble. Might there be some connection?

Another odd sign here was the state funeral in Red Square today for a deputy premier, Leonid Kostandov. According to western diplomats, no deputy premier had ever before been buried in the Kremlin Wall, and diplomats were baffled as to why Kostandov was accorded this honor.

Yesterday most of the top leaders -- but not Chernenko, or Premier Nikolai Tikhonov, Kostandov's immediate boss -- appeared at a lying-in-state ceremony for the dead man. But most of the senior leadership was absent from the reviewing stand atop Lenin's mausoleum for today's funeral service.

At the last such Red Square funeral -- for Marshal Ivan Bagramyan in September 1982 -- virtually the entire top leadership was present. Attendance at such grand state occasions by all available members of the Politburo would seem proper and normal. So why did so few leaders attend today's funeral?

In the days ahead, more information about Ogarkov's fate may leak out. Some analysts still believe it is possible that Ogarkov has not really been demoted and will be brought back in some more important post. The way his replacement was handled in the official media here would seem to preclude that possibility, however.

All that is known for certain is that the Soviet leadership has lost "a go-getter, a hard charger," as one diplomat called Ogarkov today. Those are unusual characteristics in the Soviet leadership -- a fact which, by itself, may have made Ogarkov's eventual demise inevitable.

And if there is some profound explanation for this change at the top of the Soviet military establishment, it is almost certainly not an immediate crisis. This morning, the second secretary of the Communist Party, Mikhail Gorbachov -- considered a potential successor to Chernenko -- left Moscow on a previously scheduled visit to Bulgaria. It is most unlikely that Gorbachov would have left here in the midst of tense infighting in the Politburo.