Miles Lerman, the chairman of the U.S. Holocaust Memorial Museum and one of the most visible sources of both its extraordinary success and its occasional polarizing brushfires, announced yesterday that he is stepping down from his post.

Lerman, a businessman who fought against the Nazis in southern Poland during World War II, joined the planning committees for the museum in 1978 and has been chairman since 1993. During his tenure he raised nearly $200 million to build the museum right off the National Mall. And he directed the transition from proposal to the present full-fledged museum, which has had 14 million visitors in 6 1/2 years. He was appointed chairman of the Holocaust Memorial Council by President Clinton in 1993 and will continue to serve on the museum's governing body when the White House appoints his successor.

"The tasks I set out to accomplish have been done far beyond my expectations," Lerman said yesterday. He also said the museum was running well and that his stepping down was due to his confidence in the staff, not any health problems or discouragement over recurrent criticism of his decisions and the museum's operations.

"Rotation is a healthy thing," said Lerman, who will be 80 this year though he insisted jokingly he be described as "over 51." He added, "It is time to bring in young blood and carry the standard forward."

Lerman's passion might be hard to duplicate, said Ruth B. Mandel, vice chairwoman of the council. "His leadership has never been hidden," she said. "He has been out front, outspoken and as energetic as any 55 people on the council. He has been quite clear about his understanding of the museum, his commitment, his passion and his agenda."

At times during his tenure the museum has seemed to be under siege, drawn into international Jewish conflicts and personality squabbles. Lerman was right in the middle of a flap over a 1998 invitation that was extended, rescinded and extended again to PLO leader Yasser Arafat to visit the museum, and critics accused Lerman of mishandling the whole front-page episode. In the end, Arafat declined, citing schedule conflicts. "All I wanted to do is to bring Arafat in and teach him the lessons of the Holocaust," Lerman said yesterday.

In another instance an uproar ensued when John Roth, a scholar who had compared the modern Israelis to the Nazis, was appointed to the top research job at the museum. Roth eventually declined the offer.

Last summer the National Academy of Public Administration, acting on a congressional request, looked at the internal workings of the museum and found "excessive involvement" by the council in routine operations. It named Lerman in its criticism.

"When you live in the public eye and when you live in Washington in particular, for an institution such as ours not to have criticism is unusual," Lerman said yesterday. Before the report, according to Lerman and the staff, the council had already studied some of the areas scrutinized in the report.

Not only has the museum been a draw for those interested in the Holocaust and its relationship to contemporary atrocities, but it has also helped Holocaust survivors--some of whom hesitated for decades to tell their stories--to become part of the contemporary interpretation of those times.

Lerman had always been forthcoming about the sources of his involvement. His mother, some of his siblings and almost all of the residents of his town of Tomaszow were killed. His wife survived imprisonment at Auschwitz-Birkenau. After the war the Lermans settled in Vineland, N.J., and he has centered his life outside his business career on documenting the Holocaust.

He urged that future scholars be trained through the museum's Center for Advanced Holocaust Studies and also underwrote the Miles Lerman Center for the Study of Jewish Resistance within the larger office. He was also instrumental in establishing the museum's Committee on Conscience, which through debates, forums and statements raises awareness about the modern instances of genocide. He will continue those projects, he said yesterday, as well as direct the $250 million endowment drive.

CAPTION: Miles Lerman has been chairman of the Holocaust museum since 1993.