The Carter administration, after laborious but still unfinished study, has concluded that it lacks the legal authority to comply with the conditions received from Iran Friday for release of the 52 American hostages.

According to informed sources, this was among the conclusions of a three-hour meeting of senior administration officials at the State Department yesterday on the 12-page, single-spaced document received by cable from the Algerian intermediaries the day before.

Another session of the U.S. negotiating team is scheduled for this morning, shortly before Secretary of State Edmund S. Muskie's noon appearance on "Meet the Press" (NBC, WRC). Muskie is expected to announce the first official reaction to Iran's latest proposals at that time.

As of late yesterday, the sources said, no final decision had been made about how the United States will respond to Iran's detailed requirements, which have been referred to in Tehran as that country's "final answer." The most likely course, in the opinion of U.S. policy makers, is to send back another set of Washington reactions, questions and counterproposals in hopes that the Iranian authorities will modify their demands.

"There is a long road ahead" if the administration decides, as seems likely, to continue on the present path of indirect negotiations, said an official familiar with the details. Barring unforeseen developments, there now appears to be little likelihood, under the best of circumstances, that the hostages will be released as the result of successful bargaining before President-elect Ronald Reagan is inaugurated on Jan. 20.

U.S. officials said the latest Iranian proposals contain a number of elements that probably are viewed as major concessions in Tehran, and that these indeed hold the seeds of eventual agreement on several key points.

For example, Iran has now agreed in principle to a mechanism for resolving the legal claims in U.S. courts against Iranian assets held in the United States. Though there is no meeting of minds on the details of such a mechanism, the agreement to work them out in negotiations is an important step, the sources said.

Even though he died in Egypt almost five years ago, the shadow of the former shah, Mohammad Reza Pahlavi, continues to fall across the future of the captive Americans. According to U.S. sources, the most serious problem in the new Iranian document concerns the handling of the late shah's money in the United States.

For all its difficulty, this question is of greater symbolic importance than practical significance. The former shah reportedly told White House officials during his stay in a U.S. Army hospital near San Antonio a year ago this month that he had "barely any assets" in this country. An administration official who was in contact with the shah said that in view of the political problems arising from the hostage taking, Pahlavi "would have been out of his mind to leave any significant amounts" in the United States.

After first demanding the return of the shah for the release of the hostages and then the return of his overseas forture, the Iranians now have proposed that the United States deposit large sums of money in Algeria as a "guarantee" that the United States eventually will return to Iran the American assets of the former monarch.

In an interview broadcast yesterday over Tehran Radio, Prime Minister Mohammed Ali Rajai acknowledged that "there is no accurate information" concerning the amount of the late shah's holdings in the United States. Rajai explained, according to a U.S. government translation to the interview, that Iran is proposing that "whatever amount comes to light" of Pahlavi's assets be deducted from the U.S. "guarantee" for this purpose to be held by the Algerian government.

He added that "This guarantee should not fall lower than a certain figure; if it does, the United States should raise if to its original amount."

The United States had agreed previously to undertake an inventory of the American assets of the deposed shah and not to place impediments in the way of Iranian claims against such assets of American courts. But officials said it remains out of the question, politically as well as legally, for an American president to order the impoundment of the private assets, if any, of the former shah.

Moreover, there is no way, the officials said, for a president to transfer any such assets out of the United States into the hands of a foreign power on his own authority. Any such action would require final orders of U.S. courts, which would take years of litigation.

A tangle of legal problems also persists in Iran's plan for blocked governmental assets in this country to be transferred to the Algerian government. Several billions of dollars in U.S. government hands, not encumbered by private claims, probably could be transferred readily. But it would be much more difficult to transfer assets against which private claims have been filed, according to sources.

Another problem involves the timing of U.S. and Iranian action in connection with a settlement. Rajai said in the Tehran Radio interview that "so long as the smallest part of these conditions remain unfulfilled, the hostages will not leave Iran." The united States, for its part, is unwilling to take any tangible actions until the hostages depart.

The lengthy Iranian response under study by the U.S. negotiating team, headed by Deputy Secretary of State Warren Christopher, is the latest step in a process that began Nov. 2, seven weeks ago today, with the adoption by the Iranian parliament of four conditions for the hostages' release.

After considering the conditions, the U.S. team flew to Algiers with a five-page, double-spaced reply, which the intermediaries took to Tehran Nov. 12.

After two weeks of consideration, Iran posed nine questions that the Algerian intermediaries brought to Washington Nov. 26. The U.S. answers, a 5 1/2-page, double-spaced document, were taken to Algeria Dec. 2 by the Christopher team for transmittal to Iran.

For most of this month Iran has been mulling over its new proposals, which are described here as far more detailed than anything previously exchanged between the two sides.