Economic constraints and growing competition in military sales to the Third World are complicating efforts by both the NATO alliance and the Warsaw Pact to modernize and expand their defense forces, according to the London-based International Institute for Strategic Studies, a recognized authority on world armaments.

The United States and the Soviet Union are diverting military resources from the European divide between the two alliances to the Third World, where the arms race has considerably accelerated as attempts to restrain arms transfers "have largely evaporated," the institute reported today.

In its annual review of the world's military forces, "The Military Balance," the institute concluded that "despite the continuing modernization and proliferation of military establishments around the world, the strain of limited financial and manpower resources for defense is becoming increasingly noticeable, especially in Western Europe."

In NATO countries beset with budgetary problems, high inflation, and wage, fuel and development costs outstripping inflation, military forces "will tend to shrink" even if defense spending is held steady or somewhat increased, the institute stated.

"The striking fact is that increases in defense budgets no longer produce increases in defense forces," said Director Christoph Bertram. Just keeping up with the cost of new high-technology weapons requires an average annual increase in defense spending of about 6 percent above inflation, institute experts estimate. Most NATO allies are not meeting the stated target of an annual 3 percent increase above inflation and are contemplating cuts in manpower and armaments.

"This is also happening in Eastern Europe," Bertram said. "The trend is clearly visible there, although we can't provide figures. Economic problems could be the most important impetus for arms control in the 1980s."

At the same time, the report noted, increasing competition in military equipment sales to Third World countries, for both economic and political reasons, "are causing governments to divert arms from the reequipment of their own forces, resulting in slippages in domestic procurement programs."

The Soviet Union appears to have fallen behind in supplying new tanks to the Warsaw Pact armies, according to the report, because of "substantial Soviet arms sales to Third World countries such as Iraq, India, Libya, and Syria." Since the report was completed, the United States has run short of new F16 fighter-bombers because of its sales agreements with Third World countries, most recently Pakistan.

Creation of the U.S. Rapid Deployment Force for use in the Persian Gulf area, the study noted, is diverting U.S. forces from NATO responsibilities in Europe. On the other side, "the events in Poland have underlined the longstanding uncertainty on whether Soviet planners can count on the loyalty of all Warsaw Pact members in the event of a European war. The Soviet Union may now have to divert Soviet forces to ensure the security of lines of communication and internal stability in times of war."

The institute--an independent, foundation-financed group of international defense experts--concluded that the Soviets and the Warsaw Pact otherwise continue to have an advantage over the United States and the rest of NATO in numbers of both conventional weapons and theater nuclear warheads in Europe.

It forecast that the continued proliferation of technologically advanced antitank weapons by NATO countries "is likely to complicate matters for Soviet armored forces in the future."

Because of this "revolution in antitank technology," Bertram said, experts were skeptical about the neutron shell as an antitank weapon in a European war. "We see a declining credibility in the use of short-range nuclear weapons generally," Bertram said.

The report also noted "a resurgence of interest in chemical weapons" and included for the first time a section describing and roughly estimating the quantities of chemical agents available for military use. Based on reports Bertram said were difficult to assess, the institute estimated that the Soviet Union has stockpiled 350,000 tons of chemical agents, compared to about 42,000 tons of old U.S. chemical munitions.