The Soviet Union removed its well-known chief of staff, Marshal Nikolai Ogarkov, without explanation tonight and replaced him with his principal deputy, Marshal Sergei Akhromeyev.

The wording of the announcement by the official news agency Tass suggested that Ogarkov, 67, who has long been considered a possible future minister of defense, had been demoted. Tass said he "has been relieved of the post of chief of the general staff and first deputy defense minister in connection with a move to other duties," but did not say what those "other duties" would be.

However, the impression that he had been shunted aside could be dispelled quickly if Ogarkov appears in a prominent position at the Red Square funeral of Deputy Premier Leonid Kostandov, which will probably be held Friday.

Western defense attaches here have speculated for some time that Ogarkov would be promoted to the post of minister of defense so that the present minister, Dmitri Ustinov, 75, could concentrate on his duties as a senior member of the Politburo. Ustinov is considered one of the most important members of the committee of elderly leaders now running the Soviet Union.

Ogarkov and Akhromeyev, 61, are both known for their poise and intelligence and both experts on arms control.

Ogarkov became an international figure last year when he gave an unusual press conference here to present the Soviet rationale for the shooting down of Korean Air Lines Flight 007. On that occasion he appeared forceful, intelligent and direct. He had previously made a similar impression on a number of western visitors, including members of Congress.

Ogarkov was a member of the Soviet delegation to the SALT I talks, which led to the treaty banning most antiballistic missiles and to an interim agreement limiting offensive arms, both signed in 1972. He became chief of the Soviet General Staff in 1977.

According to Tass, Akhromeyev "was born into a farming family" and has served in the armed forces since 1940, when he was just 17. He took part in tank warfare in World War II, became an officer by graduating from a military academy in 1952 and completed the course of the General Staff Academy in 1967. He then rose through the ranks to become commander of a military district and was named a deputy chief of staff in 1974.

Akhromeyev's assignments as a deputy chief of staff included heading the main operations directorate of the General Staff, a post that western intelligence specialists said included responsibility for arms control policy. In 1979 he became first deputy chief of staff and began making occasional public appearances as a spokesman on arms control issues, notably in a September 1983 press conference with Deputy Foreign Minister Georgi Kornienko on the Geneva medium-range missile talks, which the Soviets broke off two months later.

Akhromeyev had been promoted from general to marshal in March 1983 and had also become a candidate member of the Communist Party Central Committee in 1981. At a party plenum in June 1983 he was named a full member of the Central Committee, an unusual promotion that some observers saw as an upgrading of military influence in the Soviet system.

According to Soviet officials, Ogarkov was scheduled to appear on NBC television sometime during the next week. NBC is televising parts of the "Today" show and of its evening newscasts from Moscow in a special series of programs that began yesterday and will continue into next week. Akhromeyev was also scheduled to appear on NBC, the officials said.

Western observers here were baffled by tonight's announcement, and wondered whether a second, more significant shoe would drop in the next few days. Ordinarily, major changes of personnel at the top of the Soviet hierarchy are made in connection with the regular meetings of the Central Committee and of the Presidium of the Supreme Soviet, the Soviet parliament. Informed Soviet sources have said that the next meetings of those bodies are not scheduled until late October.