Sen. Sam Nunn of Georgia announced yesterday that he will not seek the 1988 Democratic presidential nomination, dashing the hopes of many conservative Democrats who considered him their party's best prospect for winning the White House.

In a letter to supporters, Nunn ended months of speculation about his intentions that had intensified following his return this week from a family vacation in Europe. "My new responsibility as chairman of the Senate Armed Services Committee weighed heavily in this decision," Nunn said. "With a son in high school and a daughter in college, I am also concerned about the impact of a presidential campaign on my family."

He said that although he was "surprised and honored" by the offers of political and financial support, "I have decided not to run in the 1988 presidential race."

"Obviously, if I were dying to be president of the United States, I'd be a candidate," he said at a news conference in Atlanta. "But I have other priorities, which have to assume first place."

Aides said he had "agonized" over the decision since last February. In Europe he pored over memos prepared by friendly "outsiders" he declined to name but who he said were upbeat about his chances of winning the presidency.

On Monday, he ordered a contingency plan, dubbed "Operation Red Light," in which aides drafted the letter announcing his intentions. He ordered it printed on Tuesday but didn't give the green light to "Operation Red Light" until Wednesday when he awoke at 3:15 a.m.

"I'm satisfied with my decision," he said, but added, "I'll be second-guessing myself for a long time to come because I considered this in all likelihood the best opportunity I would ever have in terms of being a successful candidate for president."

Nunn, 48, who is regarded as the Senate's foremost defense expert, said in his letter that the Armed Services Committee "faces many challenges in the next few months on matters relating to national security, foreign policy and arms control."

His decision was a disappointment to Democrats such as former Virginia governor Charles S. Robb and former Democratic national chairman Robert Strauss, who saw him as their party's strongest candidate in the South, a region that once was solidly Democratic but which has voted Republican in four of the last five presidential elections. Political observers of both parties agree that the Democrats must become competitive in the South if they hope to regain the White House.

Democrats officially in the race are Sens. Joseph R. Biden Jr. (Del.), Albert Gore Jr. (Tenn.) and Paul Simon (Ill.), Rep. Richard A. Gephardt (Mo.), Gov. Michael S. Dukakis of Massachusetts and former governor Bruce Babbitt of Arizona. Jesse L. Jackson and Rep. Patricia Schroeder are also likely candidates.

Democratic politicians speculated that the candidates who would most benefit by Nunn's decision are Gore and Gephardt. Gore and Jackson are the only two southerners in the race and Gephardt was the second choice of many southerners who supported Nunn.

John White, like Strauss a Texan and former Democratic Party chairman, said that if Nunn had won the nomination, the nature of the party would have been altered. "The party has lost the church goers and beer drinkers to the Republicans on the presidential level. With Nunn, we'd get the national security and blue-collar folks back," he said.

Nunn had to consider several strong political negatives, however.White and other Democrats conceded that winning the nomination could be a difficult task for Nunn, who is considerably more conservative on both domestic and national security issues than many Democrats. When the Democrats regained control of the Senate last year, Nunn realized his longtime ambition of chairing Armed Services. Some observers speculated that he had to consider the effect on his prestige and influence in that post if his presidential bid failed -- he is not up for reelection to the Senate until 1990. He also might have had trouble raising money given the size of the field. "I think it was too late to put it together and if it's too late for Nunn, it's too late for anyone," said Bob Squier, a Democratic campaign consultant. Nunn never indicated that he was burning to run. "He considered it more out of a sense of duty, that he had more to contribute than the others, than from the well-known fire in the belly," said former defense secretary James R. Schlesinger, whom Nunn called on Wednesday.

Nunn has a generally conservative voting record. He has voted for aid to the Nicaragua contras, funding for research for President Reagan's proposed Strategic Defense Initiative (SDI), Reagan's defense buildup and economic program and Reagan's nomination of William H. Rehnquist as chief justice. He has also backed efforts to overturn the Supreme Court decision legalizing abortion and a constitutional amendment requiring a balanced budget.

But he has not always been predictable. He undercut Reagan's argument that SDI could be a nuclear shield for the entire population of the United States. He also disputed the administration's contention that the Antiballistic Missile treaty allows unlimited SDI development.

He was the first to point out that Reagan's proposal to remove all nuclear weapons from Europe would leave the continent undefended and has proposed reductions in U.S. military forces in Europe to encourage the Europeans to assume a larger share of their defense.

Some Democrats feared that Nunn might have ideologically polarized the race with potentially disastrous consequences for the party.

"If it had come to be a Nunn-Dukakis, conservative-versus-liberal race, it could have gotten bloody and opened all the old party wounds," said Jody Powell, White House press secretary for former president Jimmy Carter and a fellow Georgian. "I think he'd have been a sure general election winner, but if he hadn't won the nomination he might have left the party crippled."