In a confusing end to a complicated lawsuit, the jury in the Groucho Marx estate trial today ruled against Erin Fleming, the late comedian's longtime companion, and ordered her to pay the Bank of America $471,843.

The verdict was so complex that it took more than three hours before the decision was officially announced and accepted by Superior Court Judge Jacqueline Weiss.

During that time, jurors were brought in and out of the courtroom, a court session was stopped and started and defense attorneys expressed consternation at how the verdict was being reached.

"If anyone would have loved this, the Marx Brothers would have," said Melvin Belli, one of Fleming's attorneys.

After 11 days of deliberations, a grim-faced jury of nine women and three men filed into the courtroom and informed the judge of its decision to assess $221,843.09 in compensatory damages and $250,000 in punitive damages against Fleming. The Bank of America, as executor of the Marx estate, sued Fleming for $400,000, which it claimed Fleming had fraudulently obtained from the aging comedian. Fleming was Marx's companion-secretary from 1971 until his death at age 86 in 1977.

Fleming, 42, was not in court today. Tonight, defense attorney David Sabih told reporters Fleming would not be making any statements. "She's flipping and she's going to flip more," he said.

The confusion in court began when the jury announced its decisions in "special findings"--18 questions about the case it had to answer if the jury found for the bank. When polled in open court, some of the jurors voted differently than they did in the jury room. The judge was forced to call a recess to "straighten things out."

After reexamining its verdict, the jury increased the compensatory damages by $1. Specific damages in the 18-item list ranged from a $32.75 liquor bill to $53,600 for Fleming's salary in 1976.

Defense attorneys said an appeal of the jury's decision was possible. A hearing is scheduled for Friday to consider the deposition of Fleming's property, which includes two homes and a half-interest in Groucho Marx Productions.

Jury foreman Eugene McCarthy told reporters he believed the evidence showed Fleming had broken the trust that Marx had placed in her. "We did not find any evidence that she defrauded him. What we found was that she breached the fiduciary trust that he had given her," McCarthy said.

J. Brin Schulman, the lawyer representing the bank, said, "She was something more than the classic gold digger . . . she was totally vulgar."

Belli said he was worried about Fleming. "She doesn't have a dime, not dime one. The bank will take the house she's living in, and she won't have a roof over her head. She literally does not know where her next meal is coming from."

Sabih, who presented most of Fleming's defense, has recently expressed concern that his client would try to commit suicide. Asked if that was still a concern, Sabih replied, "Absolutely."

Sabih said Fleming, who was declared "mentally ill" by a psychiatrist after several public outbursts during the course of the trial, is not currently being treated for psychological problems, but that she "has been under sedation for about a month. She's taking some pills to help keep her calm and collected."

Marx's psychiatrist, Dr. L. James Grold, testified that the comedian told him privately that Fleming was his reason for living.

"She provided a reason for him to be alive," Grold told the jury. "She would have given her life for Groucho."

Earlier this week the jury appeared deadlocked and many observers expected a mistrial to be declared. But early this afternoon the jurors told Judge Weiss that they had a verdict--but that they were going out for lunch before they would announce it. Out they went, and the verdict was announced about 2 p.m.

Fleming's legal problems began in 1977, shortly before Marx's death, in a conservatorship battle in which Arthur Marx, the comedian's son, successfully restricted Fleming's access to his father. The jurors in the estate trial heard from Marx's executive housekeeper that Marx's final days had been "saddened" as a result of that conservatorship, which placed a limit on the number of hours Fleming was allowed to visit Marx.

Other witnesses, including one of Marx's ex-wives, who is a former nurse, and such show-business luminaries as Carroll O'Connor, George Burns and Sally Kellerman also testified on Fleming's behalf. But the jury apparently put more weight on the testimony of other witnesses, including Marx's three children, who contended Fleming had abused Groucho Marx by drugging and beating him, and had used undue influence to obtain gifts and money from him.

Belli said today Fleming lost the case because of "the jury. There are nine women on that jury. Never let a woman judge another woman. And then there's that jury foreman former Santa Monica police captain McCarthy , he's a former cop. You never leave a cop on a jury . . . They are no good . . . This jury just resented Erin Fleming. She criticized, she swore, she's a free spirit, a free soul, and they're resentful of that."

Defense lawyer Sabih predicted that "Beverly Hills will be full of unemployed lawyers trying to take money away from young women who have lived with rich old men."