A massive study by the House Committee on Aging Charges that much of the cancer insurance sold in America is a "ripoff," in which policyholders typically get back less than 40 cents in benefits for every dollar in premiums paid.

The study says this is less than gamblers take away from Nevada slot machines, which usually return at least 70 cents of every dolar played.

The study, to be released today, says some insurers use misleading statistics, "scare" advertising showing pictures of famous people who died of cancer, "fine print" clauses that conceal gaps in coverage, and hidden "exclusion" provision to keep benefits low and profits high.

"Many insurers promise mountains of benefits but deliver molehills of protection," says the committee, which is headed by Rep. Claude Pepper (D-Fla.).

Cancer insurance is the major form of "dread disease" coverage. The insurer agrees to pay benefits for medical costs resulting from cancer.

According to the report, cancer insurance is the fastest growing insurance line, with 4 million policies sold in 1979 and about 20 million policies in force overall.

About 300 companies sell it, and premiums totaled about $1.5 billion last year, the committee estimates.

According to the report, American Family Life Assurance Co. of Columbus, Ga., has about half the cancer insurance market. Union Fidelity Life, American Income Life and Lone Star Life are other big sellers, the report says, and Mutual of Omaha and ITT Life are also in the market.

The committee says that while the average health insurance policy covering all forms of illness pays benefits equal to about 80 percent of premiums collected, "cancer insurers typically return about 40 cents or less of the premium dollar, keeping 60 cents for themselves in profits commissions for salesmen and administrative expenses."

According to committee figures, Mutual of Omaha and ITT return benefits equal to about 60 percent to 70 percent of the premiums on cancer policies. But some others named ranged from 19 percent for one company to 55 percent, with most a bit over 40 percent.

Lee Parker, a spokesman for American Family Life, said in a telephone interview that one of the reasons these return rates appear low is that much of the insurance has been sold only recently and people haven't held them long enough to start getting benefits. Moreover, he said, his company sells guaranteed renewable policies on which the return is always lower.

At any rate, he said, American Family Life's figure is 64 percent. "We have nothing to be ashamed of and nothing to hide," he said "The primary criticism of cancer insurance stems from ignorance. We do not use scare tactics and we don't misrepresen . . . We paid out $100 million in cancer claims last year. Those people like us."

The companies, according to the study, also exaggerate the potential costs of cancer, asserting that "that average cost of treatment is between $20,000 and $30,000" per person.In facat, according to the Massachusetts Insurance Department, the average cost for serious long-term patients is closer to $10,000.

The committee says companies frequently include wage losses and other nonmedical expenses in their cost-of-treatment estimates to scare customers as to the financial burdens but the policies don't cover these items.

Even medical expenses, the study says, fall through loopholes: In one case, a woman with bills of $4,000 for cervical carcinoma treatment was able to collect only $160, according to the committee. Some policies cover doctor bills only when the patient is in a hospital; others contain exclusions and gaps so that on an $8,000 bill involving a 40-day hospital stay, according to one study cited by the report, several policies would pay only $1,569 to $1,840.

In some cases, according to the committee, the policy contains fine print promising to pay only if the cancer is verified by pathological study (tissue examination). But the committee quotes physicians as saying that sometimes, for example, in brain or lung cancers, the method of diagnosis is nonpathological and removal of tissue for pathological verification can be life-endangering.