Brooklyn District Attorney Elizabeth Holtzman announced today that she will not seek the Democratic nomination to challenge Sen. Alfonse M. D'Amato (R-N.Y.) for the U.S. Senate in 1986.

Holtzman, 44, a former member of the House of Representatives, had been considered virtually certain to seek the nomination. Her withdrawal leaves a more open field for 1984 Democratic vice-presidential candidate Geraldine A. Ferraro, who is considering a Senate run.

In an interview, Ferraro said she will announce her decision in December. She has had meetings with state officials and national supporters for weeks. Over Thanksgiving weekend, she will lunch with Democratic Mayor Edward I. Koch and meet with Gov. Mario M. Cuomo (D), she said. Cuomo already has endorsed her.

"It wouldn't be an easy race," Ferraro said. "It would be uphill. But it's doable." D'Amato, she said, "is vulnerable on his votes supportive of Reagan's policies, which have devastated large numbers of people in New York State. He's vulnerable for his right-wing positions on social issues."

D'Amato already has raised over $5 million for his 1986 reelection bid, in part by declaring that Ferraro, an "extremist liberal," would be his likely opponent.

Explaining her decision not to run, Holtzman said, "Despite my successful efforts at fund-raising, I am not personally wealthy, and his head start means that I would have to spend most of my time in a Senate campaign raising money instead of raising issues."

Holtzman was defeated by D'Amato in 1980 in a three-way race with former senator Jacob K. Javits (R-N.Y.).

Ferraro acknowledged the difficulty of raising money in a race against D'Amato, but said Holtzman must devote time to her job, while she does not hold office and "can afford to take the time to raise money."

Asked if her husband's legal problems will affect her decision, Ferraro said, "If John were opposed to my running, I wouldn't be discussing it at all."

Other potential Democratic candidates are awaiting Ferraro's decision. Those mentioned include former governor Hugh L. Carey, former congressional candidate Mark J. Green, former New York Power Authority director John S. Dyson, and Arthur Levitt Jr., head of the American Stock Exchange.