President Reagan's announcement that the United States will not sign the Law of the Sea treaty has thrown a cloud of uncertainty over the long-term prospects for American participation in seabed mining.

Many experts believe that the American mining and petroleum companies that opposed the treaty and its provision for international control of seabed minerals may have overplayed their hand and dealt themselves out of a game they had hoped to dominate.

The stakes of the game are enormous. A lode of manganese nodules worth hundreds of billions of dollars is believed to lie three or more miles down on the ocean floor. And the recent discovery of polymetallic sulfide deposits in the Pacific offers the potential of further mineral riches, according to Amor Lane of the National Oceanic and Atmospheric Administration.

If the United States stands alone among nations in refusing to participate in the international program, these experts say, American corporations may find it legally and financially impossible to undertake the complex, expensive task of bringing up the billions of dollars worth of manganese, nickel and cobalt and copper found in nodules on the ocean floors. They say companies such as Lockheed and U.S. Steel, which have invested millions in technology and exploration, may be forced either to channel their operations through foreign subsidiaries or drop out of deep-sea development altogether.

But industry executives defended the president's decision. They said it would certainly be impossible to obtain financing under the terms contemplated by the treaty, and that in any case circumstances could change in the decade before exploitation of the seabed actually begins. They also said other industrialized nations may refuse to ratify the treaty, making it possible for the United States to reach bilateral agreements with them and go ahead with seabed ventures regardless of the treaty.

The House Merchant Marine and Fisheries Committee has scheduled hearings beginning Tuesday on the reasons for the president's decision and its implications for the mining and maritime industries. The House Foreign Affairs Committee is planning similar hearings in September.

Rep. Clement J. Zablocki (D-Wis.), chairman of the Foreign Affairs Committee, warned that "a major disappointment looms for the future of American industry in seabed mining."

The administration, Zablocki said, is hoping for a "mini-treaty" with mining states such as West Germany that would bypass the international control structure envisioned in the Law of the Sea treaty. But Zablocki and many others predict that Japan and the major European nations will sign the sea treaty. "If the United States then tries to go it alone with seabed mining ventures, it will be risking serious consequences by violating the global treaty to which others adhere," Zablocki said.

Leigh Ratiner, a partner in the Washington law firm of Dickstein, Shapiro and Morin, who was deputy chairman of the U.S. delegation to the final sea law negotiating conference, says in the current issue of Foreign Affairs quarterly that "if the United States is not part of the treaty system, American companies will have to go to other countries to be able to conduct business in the seabed. As a result, the United States will lose direct access to strategic raw materials from the seabed, a goal it has sought consistently throughout the 10-year law of the sea negotiations."

The mining industry, Ratiner said in an interview, is "fantasizing" about European nations refusing to sign the treaty and joining the United States in separate arrangements. If metals prices rise and make investment feasible, he said, it is those nations that have both valid legal claims and guaranteed sites--acquired through adherence to the Law of the Sea program--that will be able to obtain financing.

John Temple Swing, vice president of the Council on Foreign Relations and a former member of the U.S. Law of the Sea delegation, said that "a great majority of nations, probably including all our allies, are likely to accept the law. I and many others believe that U.S. companies are therefore going to wind up being shut out of the deals they sought to preserve. They may have shot themselves in the foot."

He said he could imagine "gunboat scenarios" in which treaty signatories challenge U.S. mining vessels, but "more likely is the expropriation of assets when other nations think the U.S. is violating international law."

The treaty, which was overwhelmingly approved in a United Nations vote, would give control of the oceans' assets to an international authority and an "enterprise" that would approve mining ventures, issue licenses, validate claims, set production limits and distribute profits among signatory nations. According to treaty critics, that means those who invest capital and technological know-how in seabed mining won't reap the benefits of their input.

That is why American mining interests and the administration opposed it. The argument that the United States should work within the treaty framework, however unpalatable, because American ventures cannot stand alone has not persuaded the potential seabed miners.

"I strongly supported the president's decision not to sign," said Conrad G. Welling, senior vice president of Ocean Minerals Co. (OMCO), an ocean mining consortium consisting of Lockheed, Standard Oil Co. of Indiana, Royal Dutch Shell and Royal Bos Kalis Westminster, a Dutch marine construction company.

He said opposition to the treaty "is the industry's position and it's OMCO's position. The counter arguments show the total ignorance of the other side." He said it was clear that "we cannot raise the investment capital we need under the text as written. Those who want to sign the treaty argue that we are worse off without the treaty than with it, but if you have several courses open and some say maybe while one says no, you go with the maybe. . . . Nobody is going to finance any ventures under the terms of this treaty."

The elements found in deep-sea nodules are considered either strategic or industrially essential by the United States. There is only one mine in this country where nickel, essential for stainless steel, can be extracted economically. Cobalt, used in magnets and medical devices, is in critically short supply. Copper and manganese, used to make steel, are plentiful.

Western industrialized nations and Japan have already invested more than $400 million in exploration of the sea floor. In addition, the U.S.S.R., India and China have expressed interest. Four international consortia and an all-French company have developed technology to mine manganese nodules and extract metals from them. Ocean Minerals Co., for example, has experimented with a robot mining machine that crawls along the bottom scooping up the potato-size clusters and pumps them to an intermediate buffer, which in turn sends them through an 18-inch pipe to a mother ship on the surface. They are then pumped to a cargo ship for the trip to shore, where they are crushed and processed.

The next step is to build full-scale prototypes and conduct pilot programs. With a $200 million price tag for each project, investors refuse to make commitments until the legal situation is settled and the economic situation is brighter.

Estimates by the Office of Ocean Minerals and Energy based on three exploration sites project 67 million dry tons of nodules can be raised from each 18,000-square-kilometer site. The nodules examined contain one percent or less of nickel, copper and cobalt, plus 29 percent manganese. At today's prices the four recoverable minerals are worth about $16 billion. There are perhaps 100 such mining sites in the world.

The current price of nickel is about $3 a pound; copper, 65 cents a pound, cobalt, $9 a pound; and manganese, $395 a ton. The agency's chief scientist, Robert Dill, estimates that a slight rise in market prices from their currently depressed levels would bring ocean mining to the break-even point. With nickel at $3.50 and copper at 90 cents, he said mining companies could recover three minerals at a cost of about $80 per ton, or four minerals for $130 a ton, and earn a 5 percent return on investment. A French consortium envisions annual revenues of $407 million for a company producing three minerals; $716 million for four minerals.

However, because ocean mining is such a risky venture, requiring a capital outlay of $1.5 billion to $1.8 billion per site, the industry is bound to demand a higher return on its investment before deep-sea mining becomes commercially viable.

"It's impossible for U.S. miners to make money under the treaty," said Jeffrey K. Amsbaugh, president of Ocean Mining Associates of Gloucester Point, Va., who speaks in terms of a return on investment of 20 percent or more as necessary to attract capital. Experts say a rise in the price of minerals that would make ocean mining viable depends on a host of factors, including supplies, the price of ore from land-based mines and, of course, political upheavals that could shut off sources of strategic minerals.

The best guess is that commercial mining of the ocean depths will not begin until the 1990s or perhaps the turn of the century. Given the existing legal and economic uncertainties, the industry has been on hold for the past two years. Only two of the consortia registered in this country maintain testing facilities and staff.

Whether they will continue to operate under their present structure is a subject of conjecture. A source close to the industry says that the U.S. companies are "scurrying to find ways to get under foreign flags." He added that there is already a movement abroad to form a pan-European consortium that could toss the U.S. companies out because it could not get mining sites from the United Nations if all the participants had not signed the treaty. American participants acknowledged the possibility of operating through foreign subsidiaries, but seemed more inclined to put faith in their government to work out an acceptable solution.

A 1980 law requires the United States to seek reciprocating agreements with other nations that have their own deep-sea mining operations, which is what those who oppose the treaty hope will now happen. Last week Otho Eskin, director of the State Department's Office of Ocean Law and Policy, said the effort to find reciprocal partners, who would recognize U.S. claims in exchange for U.S. recognition of theirs, would continue. But, he said, "it would be difficult to conclude such an agreement with any country that is a signatory to the treaty."

The U.S. legislation was intended to encourage American companies to undertake ocean exploration, but Eskin questioned whether "it does the industry any good. Can the industry, backed by that legislation, borrow a billion dollars? That is the question," he said.

"Can we carry out deep-seabed mining outside the treaty?" he asked. "It might very well be challenged in the International Court of Justice. We think we would have a good case there but you never can tell."

The administration, he said, will "look to creating conditions allowing companies to operate outside the treaty if they can get financing. We'll try to create as good a legal environment as possible." That could include federal loan guarantees or insurance to attract financing in the face of the risks.