The decade-old U.N. Law of the Seas Conference in Geneva has bogged down in internecine squabbling without substantive work on its agenda and faces the "probability" of failure, Special Ambassador Elliot L. Richardson said yesterday.

"We have to recognize the possibility, indeed the probability, that we won't succeed," Richardson, head of the U.S. delegation, said in an interview during a brief visit in Washington. The conference is already in the third week of a scheduled seven-week final session.

Richardson said there have been a few encouraging signs - notably a renewed determination on the part of many delegates on the part of many delegates to bridge the gulf between The Third World and the industrial nations - but he noted the conference now has little enough time in which to finish.

It's about a wash," Richardson said, reiterating his belief the conference has about a one-in-three chance of reaching agreement on the key issues, particularly on a system of governing the exploitation of vast mineral resources on the oceans' floors.

Repeatedly, Richardson sought to put as good a face as possible on the future of the conference, saying that new procedural ground rules make it less likely that contentious delegates among the 147 nations attending can scuttle the session unilaterally. But just as emphatically, Richardson offered what seemed to be pessimistic odds.

"When you say there is a one-in-three chance of success, you have to say there is a probability of failure under those circumstances," Richardson said at one point. At another, he said, "If the odds are worse than 50-50, you have to use the word 'probably' when you talk about failure."

For 10 years, the Law of the Seas Conference has been engaged in a global tug-of-war, trying to reach agreement on a treaty whose foundation is based on the principal that all resources under the sea are the common heritage of mankind.

At stake is the sharing of what could be trillions of dollars worth of potato-sized nodules of minerals lying at depths of up a 15,000 feet, mostly in the pacific between Hawaii and Mexico.

The negotiations have mired in disputes over and international authority that would control exploitation of the rich deposits of nickle, cobalt, manganese and copper, and manage the distribution of profits among nations.

Generally, the Third World has favored the creation of an operating body, called "The Enterprise," that would harvest the United States and other industralized nations favor a dual system under which "The Enterprise" and private mining firms would operate simultaneously.

But the conference has not even reached those issues as it approaches the midway point.

For two weeks, delegates fought bitterly over the election of a president. Shirley Amerasinghe of Sri Lanka, and a number of Latin American nations staged a brief walkout in protest. Then a delegates began fighting over procedures and the wording of the agenda, prompting Americasinghe to complain, "We will look ridiculous if we go on like this."

Displaying increasing frustration, Richardson told the conference he did not see how substantive negotiations would be influenced by the wording of an agenda, and then he headed home for previously scheduled testimony before the House Ways and Means Committee.

In yesterday's interview, Richadson said the conference conceivably could adjourn at the end of the seven weeks and resume next year, providing it was close to agreement and only peripheral issues remained unresolved.

However, many observers believe the present session is a last chance to set seabed mining guidelines, and Richardson previously has said as much. Several consortiums of mining firms already have begun experinmental seabed mining, abd a few are poised to begin fulllscale operations, having invested up to $50 million each in research and development.

Richardson said, "it would only be

If there is to be another session, justified by demonstrating that enough progress had been made in this session to leave only residual points to be cleared up."

Apart from the question of seabed minerals, one of the most difficult issues the conference faces is the right of access of landlocked and "geographically-disadvantaged" nations to predetermined "economic zones" in the oceans.

Diplomatic sources said, in fact, that this issue has nearly eclipsed the seabed mining controversy in the minds of some of the delegates from developing nations.

Once the United States accepted a dual system of exploitation, the conference sources said, the Third World seemed satisfied that they had established several basic principals pointing vaguely to a new economic order: world acceptance of the principal of common heritage of resources, acceptance of international authority and a socialist "Enterprise" and acquiescence by industrialized nations to at least some level of transfer of tecnology to lesser developed nations.

As the conference continues, the House is working on a bill that would authorize and encourage U.S. mining companies to unilaterally begin seabed mining operations.

Conference sources said somer delegates have been warning that if the bill is passed during the Geneva session, it would destroy the chances for a treaty. But, it is unlikely the Senate will act before then.