A jury in Los Angeles has dismissed a lawsuit accusing the 11-million-member American Association of Retired Persons and the Colonial Penn Franklin Insurance Co. of fraud in connection with a health insurance policy recommended by the non-profit organization to supplement Medicare.

The jury voted 10 to 2 Friday to reject allegations by AARP member Anne M. Costello that the company and the association had fraudulently induced her to buy an "in-hospital" policy and then breached a duty of good faith by refusing to pay a claim.

She had spent 98 days in one of the nine general hospitals in Los Angeles, including 59 days in emergency surgery, in an iron lung and in other forms of intensive care. The total bill was $21,000. Costello sought $3,000, the sum she thought would be paid under her policy.

The company refused to pay the $3,000, saying that the hospital wasn't within the terms of her policy, although it has been accredited since 1957 for both acute and rehabilitative treatment. After she sued, however, the company paid the $3,000 -- but said it did so purely as a matter of good will.

The company's press spokesman couldn't be reached.

David I. Lipsky, an attorney for Costello, said that several jurors, in post-verdict interviews, told him they concluded that the company made a mistake in initially refusing to pay but had not shown bad faith.

They also said that at one point during nearly three days of often strenuous deliberations they voted 7 to 5 for Costello, Lipsky reported. He said the jury never reached the question of punitive damages, which she had sought in an unspecified amount. An appeal is possible, Lipsky said.

Costello named the AARP as a codefendant because in exchange for specified sums paid by the company's owner, Colonial Penn Group of Philadelphia, the association and its smaller affiliate, the National Retired Teachers Association, mention and promote to their members only CPG's health, life, auto and home owner policies.

Last January, after a 2 1/2-year investigation, a Postal Service investigator recommended revoking the association's non-profit mailing privileges. The recommendation is pending.

On Dec. 20, in the trial's second day, Colonial Penn moved for a mistrial Out of the hearing of the jury, Superior Court Judge Francis X. Marnell denied the motion, saying that the evidence already showed an on-the-face-of-it fraud, and that a jury could conclude that the fraud was "malicious."

Emphasizing that he would not tell the jury of his opinion, Marnell said that Colonial Penn, in saying that Costello's hospital care wasn't covered, gave "the most facetious reason I've ever heard for turning down a claim."

Also with the jury out, plaintiff's witness Robert E. Schultz, a consulting economist, told the judge that Colonial Penn's profit situation, for reasons he did not know, was "truly unique" in the health-insurance field.

Most health insurers would find a before tax profit of 5 percent "very acceptable," knowing that anything higher "would bring in a great deal of competition," Schultz said. But with its exclusive access to AARP's members, Colonial Penn has increased its rate of return year after year and in 1977, when it had a profit of $51,853,000 on premium income of $261,280,840, the rate was about "four times... normal... for this type of business," Schultz said.

The company's "in-hospital" policy, offers payments starting with the eighth day in a hospital. Often, experts say, the types of treatment covered end with the sixth or seventh day in a hospital.

Of each premium dollar collected in 1977, Schultz said, Colonial Penn paid back to policyholders 53 cents in California and 62 cents in the nation as a whole.