A $54 million, five-year-old federal program to show the value of X-rays in detecting breast cancer would not be started today without harder evidence that the X-rays find more cancer than they cause, a government cancer official said yesterday.

Dr. Diane Fink, director of cancer control for the National Cancer Institute said the X-ray, or "mammography," project was started in the first flush of a federal anti-cancer crusade, when the understanding of X-ray risks was "a bit fuzzier."

Fink's statement was the start of the latest review of the controversial program that so far has given annual exams, including mammograps, to more than 200,000 American women. The review is being made in a three-day meeting of the nation's leading doctors and scientists, at the National Institutes of Health in Bethesda. The aim of the meeting said Dr. Donald Fredrickson, NIH director, is to reach a "consensus" on whether or not to continue the program, and if so, how extensively.

A committee headed by Dr. Oliver Beahrs of the Mayo Clinic will make its recommendation today. No conclusion will be reached before Friday, and perhaps not then unless Fredrickson gets a clear signal one way or the other from the group.

The 200,000 women enrolled in the program - down from nearly 280,000 at the outset - get annual exams including a mammogram at 27 major medical centers, including Georgetown University.

The American Cancer Society, which co-sponsors the project, played a major role in pushing the Cancer Institute to start it.

Dr. Benjamin F. Byrd, head of the Cancer Society's breast cancer task force, said yesterday that mammography has helped turn up 2,500 cancers in the study group and has proved itself as "the significant modern advance in detecting early breast cancer."

But several other speakers agreed that it still has not been shown how many lives will really have been saved by adding mammography to other examinations - and how this will balance out against deaths caused by excess radiation.

Dr. John Bailer, editor of the Journal of the National Cancer Institute, said that starting about 10 years after the first x-rays there will be 3.5 to 7.5 extra breast cancer deaths each year for each million "rads" of x-ray exposure. The women in the project's first year alone may have received nearly that much radiation, and that risk is multiplied by the number of years in which each woman is exposed.

Annual X-rays in the program are now recommended for only two groups that experts so far agree have an especially high risk of developing cancer: women over 50, and women between 35 and 49 who have either had cancer in one breast already, or who have a mother or sister who developed breast cancer.