An agreement being negotiated between lawyers representing American and Iranian banks is the key to freeing up the bulk of the funds the United States proposes to transfer when the 52 American hostages are released, informed sources said yesterday.

With time running out before the Friday deadline set by the administration, a successful conclusion of the bank negotiations -- which would free up about $4 billion in frozen Iranian assets -- adds one more imponderable to the already complicated attempt to achieve release of the hostages.

"Without the cooperation of the banks," a top Carter aide said yesterday, "it will be impossible to get this thing done."

Meanwhile, Deputy Secretary of State Warren Christopher and his team in Algiers continue to work on the complex executive agreement between countries that will govern the hostage release, along with the details of an escrow account that would hold whatever funds are available before Jan. 20.

Both the Americans and the Algerian intermediaries also are waiting for some sign from Tehran on just what the Iranians are willing to accept. With the deadline for completion of negotiations just a day away, the response will have to be almost complete acceptance of the last American proposal if a deal is to be consummated by Jan. 20.

While agreement appears to have been reached on other conditions for release of the hostages -- setting up an international commission to handle claims and a plan to seek return through the courts of whatever assets of the shah and his family are in the United States -- the amount of frozen assets Iran wants returned before it lets the Americans go has yet to be established.

Secretary of State Edmund Muskie told a news conference in Augusta, Maine, yesterday, "What is at stake here is the question of how much we can make available, whether that is sufficient to achieve release of the hostages and whether or not we can put in motion a procedure for clearing the remainder."

Immediate return to Iran of a significant portion of its frozen assets has always been a key provision of any U.S. plan to end the hostage crisis.

The only Iranian assets under direct control of the U.S. government government and available for quick transfer in any hostage deal are those held in the Federal Reserve Bank of New York. These consist of about $1.3 billion in securities and 1.6 million ounces of gold worth roughly $1 billion.

A second category of frozen assets are the Iranian bank deposits that were in U.S. bank branches in London and Paris on Nov. 14, 1979, the day President Carter ordered the assets frozen.

A third category of Iranian assets in the United States consists of bank deposits from Iranian government entities, deposits in a Pentagon trust fund, weapons and spare parts already paid for, accounts payable on the books of American corporations (such as U.S. oil companies that bought Iran's oil and received it after the freeze), and material purchased from American companies and not delivered.

Because legal claims have been filed against this last category of assets, it would take weeks, if not months, to free them for transfer to Iran. As a result, the Carter administration has told Iran it must be satisfied with a guarantee that once legal questions are settled, these assets, too, will be returned.

Neither country has a common list totaling the assets. Iran at one point claimed they totaled $14 billion. American officials have put the number publicly at "above $8 billion," and privately at about $11 billion.

Recently a figure of $9.5 billion in total frozen assets has been used by Washington officials, who then went on to say that the United States was prepared to return $7.3 billion at the time of the hostage release. Such estimates of the figures involved were based more on guesswork than on actual facts, one participant in the talks said yesterday.

Within the past few days, a representative of Iran said he had been asked by Tehran officials to supply a list of the assets frozen in this country. And an official working on the negotiations conceded yesterday that, despite attempts by the U.S. Treasury to keep track of the assets, the "figures we have are very loose."

Representatives of a dozen American banks and the Iranian central bank have been meeting in New York, London and Paris for almost a week attempting to reach an agreement.

At the time of the freeze, Iran had more than $4 billion on deposit in the European branches of these banks. As part of the freeze order, the banks were permitted to take from those frozen deposits an amount equal to the loans outstanding to Iranian government borrowers that were forced into default by the freeze.

Now, the negotiators are trying to restore the situation to that which existed on the day before the freeze. Doing that, however, is complicated by more than just reinstating accounts, determining interest payments and reestablishing loans.

One complication is a political one. It is uncertain what authority the Iran representatives, who are American and European lawyers, have from the Iran government. There is a conflict in Tehran over just who controls the Iranian central bank -- part of the bitter infighting for power between Iran's prime minister, Mohammed Ali Rajai, and the country's president, Abolhassan Bani-Sadr.

A second complication, affecting the American banks' position, is what collateral the banks will have for the nearly $1 billion in loans that are to be reinstated. Many of the entities that took out the loans under the regime of the late shah are no longer in existence.

A measure of the caution involved is that the U.S. government has taken a hands-off position on the bank discussions. Officials say no effort has been made to force an agreement despite the fact that a solution to the hostage question could hang on release of the frozen European money.

One official suggested, however, that if the bank agreement is not reached in time, the Iranians would have to be satisfied with a presidential order that their deposits would be released once the banks' accounts and loans are returned to their pre-freeze situation. The Iranians have turned down the promise of such an order in the past.