New, less rigid leadership at the top of the Soviet defense establishment has paved the way for the highest military men of both the United States and the Soviet Union to sit for the first time at a summit negotiating table, according to U.S. sources here.

Since the Soviet military plays more of a key role in establishing its country's strategic arms control policies than does its American counterpart, the opportunity for President Carter and others to speak directly to Soviet Defense Minister Dmitri Ustinov and the chief of the Soviet general staff, Marshal Nikolai Ogarkov, is viewed as a major achievement by the sources.

It takes on added importance because of the prospect that Ogarkov, 61, will retain power long after ailing Soviet President Leonid Brezhnev, 72, has passed from the scene.

President Carter, in his statement to Soviet President Leonid Brezhnev this afternoon, noted that the two countries' highest miliary leaders had not met since shortly after World War II. The implication of the president's statement, according to informed sources, was that further meetings between top defense officials should be held.

Ustinov, 70, has been a major figure in Soviet defense policy since the 1950s and is known to be more flexible on the issue of detente than his predecessors.

Ogarkov, who was the Soviet Ministry of Defense representative on the first Strategic Arms Limitation Treaty (SALT) negotiating team in 1969, is described as a military man who "recognizes some decisions must be made in a political context," according to a U.S. official who knew him at the time.

Ogarkov last winter made a strong favorable impression on a group of House Armed Services Committee members who met with him for three hours in Moscow. They were impressed with his cool, reasonable delivery of Soviet military policies with which they strongly disagree.

It is an irony of today's meeting, that the very military men who have helped direct Soviet development of strategic ICBM systems that threaten U.S. land-based missiles are seen by both military and diplomatic observers as the main hopes for arms control advances in the future.

Unlike U.S. military leadership, which changes often at both the Defense Department and the Joint Chiefs of Staff levels, the Soviet system is rooted in continuity.

The long experience of the men who reach the top of the Soviet general staff and Defense Ministry gives their views on policy matters additional weight.

That influence has been particularly evident in the Soviet side of the SALT negotiations.

While the U.S. SALT delegation, from the start, has been dominated by presidential appointees and professional diplomats, the Soviet military has had a major role not only in making arms control policy but also in working out those positions with the U.S. delegation. Ogarkov, for example, ranked second only to the chief of the Soviet delegation at Geneva.

For Soviet military men, service on the SALT negotiating team has often led to promotions. On the American side, in contrast, general officers representing the Joint Chiefs of Staff have frequently been about to retire.

After leaving the Soviet SALT team in 1970, Ogarkov was promoted to first deputy defense minister under Ustinov and then to chief of the general staff. He is considered the most influential of the three first deputies.

At this juncture of the arms race, the United States is poised to build a new, mobile, land-based ICBM called MX and the Soviets are capable of constructing a similar system.

Since the purpose of such mobile systems is to make them impossible for the other side to destroy, serious questions have been raised of how in the future their numbers could be verified under the SALT II treaty which will be signed Monday.

American officials originally had hoped that Defense Secretary Harold Brown and Gen. David Jones, chairman of the Joint Chiefs of Staff, would have an opportunity here n Vienna to describe in some detail how verification of the MX could be accomplished.

Such a complex discussion, it was felt, could not be undertaken during the limited time available for talks between the two presidents.

At this afternoon's summit session, Brezhnev made reference to verification problems associated with the MX basing.

According to Brezhnev's spokesman, Leonid Zamyatin, the Soviet leader said, "As the basing takes place, we will see how it accords with SALT II."

If the MX "proves beyond national means of verification," Zamyatin continued, "it would be a mine planted under future negotiations."

White House Press Secretary Jody Powell added that President Carter responded that the MX as deployed would be verifiable under the treaty.

That Carter-Brezhnev exchange, which came as part of longer presentations by the two leaders, seemed to set an outline for the very military talks American officials had hoped to see take place.

At the first SALT talks, according to one source, Ogarkov showed himself to have a quick eye for inconsistencies or weaknesses in the American negotiating positions. He gained the wary regard of the Americans, who found his manner cool and correct, but engaging. His perspectives seemed wider than some other Soviet delegation members. Ogarkov is now a full member of the Central Committee of the Soviet Communist Party.

Western military experts closely link the rise of this sallow-faced career officier to patronage from Ustinov, who himself owes his post as defense minister to Brezhnev.

Trained as a military design engineer, Ustinov has wide experience in managing the little-known Soviet military arms establishment. Nominally a civilian (he held a wartime generalship), Ustinov in the 1950s and 1960s was identified closely with the development of Soviet rockets.

Ustinov was appointed defense minister in 1976 and was elevated to full membership in the Politburo.