A federal judge in Texas ruled yesterday that former president Carter exceeded his lawful powers in ordering the return to Iran of its now-frozen assets as part of the hostage release agreement.

The ruling by U.S. District Court Judge Robert W. Porter all but guaranteed that a Supreme Court test of the hostage agreement will be necessary before the second transfer of the Iranian funds -- some $2.2 billion held in American banks -- will take place.

The Carter administration officials who negotiated the agreement expected it to be challenged in court and included provisions giving the Americans up to six months to deliver frozen bank funds.

The officials told the Algerians, who served as middlemen for the Iranians in the negotiations, that they expected to defeat all the court challenges.

There are 388 suits against Iran in federal courts all over the country in which American claimants are seeking payment for alleged losses of property or breaches of contract. As part of that legal process, the claimants have sought legal orders, called attachments, against the Iranian assets frozen in this country.

While the Reagan administration has carried on its well-publicized review of the legality of the Carter-negotiated agreement, the Justice Department has gone into court asking judges such as Porter not to take further legal steps until Feb. 27, by which time the new president's position on the remaining provisions of the agreement is supposed to be decided.

Under the agreement, Iran's still-frozen financial assets would be transferred to the Federal Reserve Bank. From there, the first $2 billion would be divided, and one half would go to Iran and the other half would go into a special account in the Bank of England to be used for the repayment of American claims as decided by a yet-to-be-established international commission.

Many of the commercial claimants have voiced dissatisfaction with the agreement because there is no certainty about just how the proposed claims commission would operate. Government lawyers have reassured the claimants in private sessions, however, that they will not have to drop their court claims until after the proposed commission has agreed to consider them.

In yesterday's action, Porter granted a temporary injunction sought by the Dallas-based Electronic Data Systems Corp. (EDS), a giant computer firm, to block the transfer of $20 million in Iranian funds now held by Marine Midland Bank of Buffalo, N.Y.

The injunction, if upheld, would prohibit the bank from turning those funds over to the Federal Reserve Bank of New York, as required under an executive order signed by Carter on Jan. 19.

On Jan. 21 U.S. District Court Judge Gerhard Gesell ruled against a similar effort by a claimant to get a temporary injunction against Iranian assets, saying that Carter was within his powers to undertake the agreement and nullify any attachments against the frozen money.

Porter declared yesterday, however, that Carter's executive order "raises serious constitutional issues with regard to the power of the Executive Branch to nullify" court orders. He also said Carter "was without statutory or constitutional authority to order the vesting of foreign assets in the custody and control of the Executive Branch."

The Carter executive order may be "without legal effect" because it took effect" because it took effect after Carter left office and was not signed by President Reagan, Porter added.

EDS, whose founder is H. Ross Perot, won a lawsuit against Iran in May 1980, when Porter decided the Tehran government had defaulted on its contract with the Dallas company which had been hired by the regime of the former shah to computerize the national health and social security program and the office of the shah's wife.