As "a society ravaged by heart disease" and early death, America can't afford not to adopt a lower fat, lower cholesterol diet, a leading government doctor said yesterday in reply to recent advice to the contrary.

Dr. William Castelli, medical director of the famed Framingham, Mass., study of life-style and disease, appeared at the National Academy of Sciences alongside a consumer group that feels the same way.

Castelli said he is "distressed" and the Consumer Liaison Panel of the academy's Food and Nutrition Board said it is aghast at the board's May 27 statement suggesting most people can stick with their present diets.

James Turner, panel chairman, announced that it is severing relations with the board, which it unsuccessfully advised on this issue. The panel will continue to work on its own, he said, and hopes to be asked to maintain a tie with the science academy.

He urged the academy to broaden the food board's membership, in part to avoid too much food industry influence.

The food board said May 27 there is no firm scientific evidence that most healthy persons need to eat less fat and cholesterol to try to avoid heart attacks.

The board's advice is "illogical," "inconsistent" and "unwarranted," Castelli said. He gave two main reasons: (1) the food board said Americans should eat less salt, lose weight and exercise, but there is no firmer evidence backing this advice; (2) there are indeed many human and animal studies backing diet changes to combat heart attacks.

The board said some people should consider diet change, Castelli pointed out. But these people -- the overweight, those with family heart disease histories, those with high blood cholesterol levels -- add up to well over half the population, he said.

He told how the Framingham investigators -- sponsored by the National Heart, Lung and Blood Institute -- have followed 5,000 men and women for 30 years. In study's first 14 years, Castelli said, "every eighth man 40 to 44 had a heart attack," and every sixth man 45 to 49, every fifth man 50 to 54 and every fourth man 55 or older.

"We know what's happening in Framingham is happening all over this country," he said.

And a "prudent" diet, he said, should include fewer fats generally less meat, unsaturated (usually soft or liquid) rather than solid, saturated fats, no butter, "only two eggs a week" and more reliance on vegetables, legumes (like peas and beans), grains and other complex, nutritious carbohydrates, rather than simple ones like sugar.

Turner, a lawyer and author of "The Chemical Feast," an indictment of food additives, said he does not doubt the honesty, integrity and ability of a food board member like chairman Alfred Harper, a University of Wisconsin scientist who earns 10 percent of his income as a food industry consultant.

What is needed is "balance," Turner maintained, so "we're saying these people shouldn't so much be replaced as added to." Also, he said, the associations of everyone making public recommendations should be made clear.

Turner said that in 1978 the consumer panel helped prevent the Agriculture Department from picking the "change resistant" food board to review the Senate Select Nutrition Committee's "Dietary Goals," goals that this year led two federal agencies to make similar recommendations.

Now, Turner said, "we'll go to the National Institutes of Health," Congress and other funding agencies to tell them to shun board advice unless it is revamped.