Some key U.S. banks have begun talks on reestablishing ties with Iran and setting a quick return of its frozen assets, informed sources reported yesterday, amid continuing signs that the United States and Iran are making a desperate effort to reach an agreement for release of the American hostages before the Friday deadline set by President Carter.

The banks involved in the negotiations hold more than $3.5 billion in frozen Iranian accounts in branch offices overseas. They have begun meeting to determine how they would reestablish with Iran the banking relationship that existed before the freeze began more than a year ago and set the stage for quick return to Iranian control of the now-frozen accounts, sources said yesterday.

At the same time, it was learned, Iranian officials are inquiring about how U.S. banks could transfer money from their branches in Europe to banks in Algeria.

In Algiers, Deputy Secretary of Warren Christopher and his negotiating team held an urgent meeting yesterday with the Algerian foreign minister to discuss new ideas that had been forwarded to him from Washington. Christopher's group announced it was staying in Algeria "as long as necessary," according to news agencies.

In Tehran, Ahmed Azizi, an official from the prime minister's office who had played a key role in the hostage negotiaitons, told reporters that progress in the negotiations was relatively good and that he is optimistic that a solution will be found before Carter leaves office.

U.S. officials have told the Iranians that an agreement must be reached by Jan. 16 for Carter to approve it before he surrenders his office to Ronald Reagan Jan. 20.

The United States has promised to turn over the $5.5 billion to Iran upon release of the hostages. This would be made up of $1.5 billion in securities held by the Federal Reserve Bank in New York, 1.6 million ounces of gold also held by the bank and -- once agreement with the U.S. banks is reached -- the $3.5 billion in assets frozen in the overseas branches of those banks.

Iran has claimed the United States froze $14 billion of its assets. The United States publicly acknowledged the figure is "more than $8 billion," but officials privately put it at between $11 billion and $12 billion. Roughly $6 billion has been listed in American court suits aimed at attaching that money as repayment for claims against the Tehran government by U.S. corporations and individuals.

The flurry of activity by the U.S. banks and the Iranian governments in gathering details on various aspects of the complex frozen-assets problem indicates that current negotiations are focusing on the return of the assets.

Iranian officials, for example, last week reportedly began to put together their first detailed list of the frozen assets inside the United States, including what they consisted of, where they existed and whether any legal claims were being placed against them. Up to that time, Iran's list of frozen assets had been complied from information on hand in Tehran.

Release of Iran's frozen assets is, however, only one of four conditions for the hostages' release first defined last September by Ayatollah Ruhollah Khomeini and confirmed in November by the Iranian parliament.

The two sides reportedly have still not agreed on Iran's demands that hundreds of legal claims by U.S. companies and individuals be dropped and that the wealth of the late shah and his family allegedly transferred to the United States be restored to Iran. Less than a week ago, White House sources said no legal way existed for the United States to meet the Iranian demands on the shah's assets.

The final condition, a pledge that the United States not interfere any more in the internal affairs of Iran, is no longer considered a problem.

Since Carter froze Iranian assets on Nov. 14, 1979, U.S. banks have been permitted by a Treasury Department regulation to use the deposits to offset any Iranian loans they had outstanding and which then were declared in default.

The current discussions by American bank representatives are designed to set up a mechanism for implementing provisions of Iranian and American proposals that already have been exchanged on the subject.

An outline for the banking agreement is contained in Iran's note that was received in Washington on Dec. 19. It calls for the U.S. banks to nullify their loan defaults. In return, Iran proposes to put $1 billion of its unfrozen funds in Algerian hands to be used to pay delayed installments due on those loans.

For its part, the United States proposed in its December message to Iran that it would order the reinstitution of Iran's accounts by the banks that had used the frozen money to set off loans once the banks and Iran had restored their banking relationships. At the same time, the United States said it would order those accounts transferred back to the control of Iran.

"The legal questions involved are enormous," one participant in the process said yesterday, "and the transactions contemplated are very complicated."

The process by which Iran is attempting to put together its own list of its assets now frozen in the United States is also slow and time-consuming. Its representatives must contact all the banks, corporations and other institutions known to have Iranian assets and get a complete description of the assets. They also must discover which of the assets have been subjected to legal claims.

At least one of the frozen-asset lists being prepared at Tehran's request contains military equipment that has been paid for by either the current government or governments before it. No specific request was made for the equipment by the Iranians, but it is being included as one of the assets still held by the United States.

One illustration of the problems that still may exist in just determining the frozen assets is an apparent discrepancy between what Iran believes has been frozen in the way of oil payments and what can be found in the United States.

Iran maintains that at the time of the freeze, it had oil worth about $2.5 billion being transported to the United States. U.S. customers then owned Iran for the oil received. Because of the freeze, however, the payments to Iran could not be made.

Iran's representatives reportedly have been unable to locate the funds due for those oil shipments. The U.S. Treasury's public accounting of Iran's estimated frozen assets show only $500 million being held by American companies -- one category that could be made up of frozen payments for oil shipments.