Private American investors and the U.S. military stand to lose hundreds of millions of dollars if Third World and other delegates to an international radio conference here succeed in efforts to reassign key frequencies used for sophisticated defense and business communications.

In a demonstration of their concern, U.S. delegates summoned representatives of NATO countries here to a meeting last week to discuss the possible loss of frequencies currently used for the U.S. early warning system that helps protect the United States and its allies from enemy missiles.

Informed sources said the Americans were upset with their NATO allies for their apparent failure to support U.S. propsals that would allow the Defense Department to continue using several radio frequencies.

Although the loss of the defense communications would not cause a security problem for the United States, one source said, the Defense Department would have to spend "hundreds of millions, maybe a billion, dollars" to their existing equipment for use on other frequencies.

Furthermore, some sensitive military communications services could be affected. Some of the ground and mobile airborne radar equipment the military is trying to protect is so secret that civilians in the 65-member U.S. delegation have not been told the nature of the systems.

The military problems, however, make up only a small part of the complex issues being debated at the current World Administrative Radio Conference here. The financial stakes are even higher for the billion-dollar U.S. communications industry.

Many developing countries are seeking a greater share of the radio frequencies that the conference assigns. These states outnumber the industrialized nations represented at the meeting and complain that the world's airwaves are dominated by a relatively small number of advanced countries.

The conference, which began Sept. 24 and is scheduled to end in December, has brought together 2,000 delegates from 150 countries to coordinate the use of the radio spectrum through the end of the century. The conference is sponsored by the International Telecommunications Union, a specialized agency of the United Nations.

One U.S. a delegate said it was "astonishing" that the NATO countries were not supporting U.S. proposals to keep the same frequencies used by some American defense communications systems.

Sources said it appeared that many NATO countries did not wish to confront developing countries seeking the disputed frequencies for other purposes. Many of the NATO countries are said to believe that they might have to make concessions in other areas if they support the United States on the defense issues.

A final decision on the issues is not expected for weeks, but observers say it is clear that the United States will lose at least some radar-related frequencies by the end of the conference.

Although U.S. business interests seem to be faring better than military ones at the conference, some key industry proposals also are having problems.

In one case, several large American companies are trying to protect big investments in new business communications satellites that are to be shot into a special orbit whose use now is being debated at the conference.

At issue is the geostationary orbit 22,300 miles above the equator in which satellites remain over the same spot on earth as they orbit at the same speed that the planet rotates.

One firm active in the debate is Satellite Business Services, owned partly by IBM, Comsat and Aetna. It plans to launch the first of three satellites -- each costing about $400 million to put up -- into this orbit next year. Western Union, American Telephone and Telegraph and RCA plan to follow.

Some alternate plans for that satellite spectrum, however, could sharply reduce the number of orbital slots available to U.S. industry in favor of more generalized uses by other countries, notably Canada. Such plans could give major headaches to the U.S. Federal Communications Commission, which has only approved the initial Satellite Business Services launching and which could be faced with having to decide which companies can or cannot provide what is expected to be an extrememly lucrative business.

Services such as high-speed transmission -- allowing perfect copies of documents to be transmitted from one city to another at the rate of 30 pages per minute, for example -- are envisioned as part of this newly proposed plan.

Although delegation sources hope an acceptable compromise in this area can be reached, the possibility remains strong that the issue could be postponed to a specialized conference four years from now, forcing several U.S. firms to gamble billions of dollars that they will not be force out of orbits they launch into.

Previous radio conferences -- the last one was in 1959 -- were devoted almost entirely to technical matters and were dominated by the industrialized nations. There was virtually no problem assigning frequencies because there was plenty of room on the radio spectrum.

Now things have changed. The developing nations have since begun to initiate more of their own broadcast and telecommunications services. As a result the most popular frequencies -- those that offer the least interference or the best range or require the least power to transmit over -- have become more and more crowded.

Describing the airwaves as an international resource and concerned that the industrialized countries might grab the most attractive frequencies many developing countries are now proposing that some slots remain unused until they develop or can afford the technology needed to use them.

The United States feels this could lead to inefficient use of the airwaves, and stymie growth of new technology. Instead the United States wants to find ways to accommodate new entries as they come along, usually by implementing technologically advanced means for more users to share crowded frequencies.

The U.S. delegation and its chairman State Department official and law professor Glen Robinson, have received high marks from many Third World delegates for recognizing the new political realities of radio spectrum spectrum management.

"The U.S. is bending over backwards to negotiate and take a conciliatory attitude," said Sam Butler of Liberia. He suggested that U.S. business interests will do better than the military ones because proponents of the latter have been secretive while industry spokesmen have gone out of their way to be open and take their case for more complex, but more equitable, frequency allocation to anyone who will listen.