A federal judge in Miami, in a major blow to government efforts to deal with the problem of ailing thrift institutions, yesterday ruled that the Federal Home Loan Bank Board had illegally seized Biscayne Federal Savings and Loan.

U.S. District Judge Eugene P. Spellman ordered that the troubled thrift institution be returned to its former owners. He gave Biscayne's shareholders and the federal regulators 30 days in which to work out a settlement between them.

If they fail to reach agreement Spellman ordered both sides to submit proposals for returning the savings and loan to its stockholders. He set a Nov. 24 hearing date on the issue. On four other counts in the stockholders' suit, Spellman ruled in favor of the Bank Board.

Despite government assurances at the time of the takeover last April, there was a run on Biscayne resulting in a $40 million withdrawal the following day. Much of the difficulty in restoring the thrift to the stockholders will center on how to return it to its previous condition.

There are approximately 1,500 Biscayne shareholders. In anticipation of a favorable decision, the stock has recently been trading over the counter at $1.25 per share. Prior to the takeover, it was worth $5.375 per share.

Spellman withheld a final order until a remedy is devised. Until then the government will continue to run the thrift as New Biscayne Federal Savings and Loan. Board chairman Edwin J. Gray assured depositors their savings would continue to be federally insured in the interim.

Ronald Kabot, Biscayne's chairman and a senior vice president of the Los Angeles real estate and insurance firm, Kaufman & Broad, which owns 25 percent of Biscayne's stock, declared, "I am absolutely delighted by the decision." A spokesman for the Bank Board declared there was a "strong likelihood" the decision would be appealed.

This marked the second time in two years that the courts have found that federal regulators used unwarranted means in seizing an ailing savings and loan from its stockholders. The first suit involved Fidelity Savings and Loan Association of Oakland, Calif., in June 1982. An appeals court later reversed the decision and the case was sent back to a lower court to determine whether the government had wrongly declared Fidelity insolvent when it still had assets. The lower court decision is still pending.

Biscayne, Florida's fifth largest thrift, had assets of $1.9 billion but negative net worth when the regulators declared it insolvent and placed it in receivership April 6. The shareholders, whose stock had a market value of $10 million, found themselves wiped out.

They sued the government, charging the government had exceeded its authority because there was no emergency. During the five-week trial, Biscayne's attorney, Bruce Greer, repeatedly accused the Bank Board of deceiving Biscayne's stockholders by engaging in negotiations with Kaufman & Broad to recapitalize the thrift while at the same time planning to take it over.

The non-jury trial was unusual in that the Bank Board's chairman at the time, Richard T. Pratt, and the Federal Savings and Loan Corp. chief, Brent Beesley, were required to testify in person to explain their actions. Beesley testified that he was under orders from Pratt to "get the best deal" he could. That deal called for Kaufman & Broad to infuse $38 million in capital and the FSLIC to put in a nonrefundable $25 million.