At his news conference Jan. 7, President Reagan denounced Libyan leader Muammar Qaddafi as "flaky" and "a barbarian" and said he "deserves to be treated as a pariah in the world community." At his news conference last week, the president referred to Qaddafi as "this mad dog of the Middle East."

This dehumanizing discourse is not likely to cost Reagan any popularity points with the American people. It is difficult to overstate the case against a national leader who openly endorses terrorism and praises the killing of unarmed civilians. But it is even more difficult to discover a useful policy purpose in the president's name-calling.

Privately, some officials acknowledge that Reagan's verbal efforts to tear down Qaddafi have instead succeeded in building him up in the Middle East. They share the point of view of such outside critics as retired admiral Eugene J. Carroll, who once commanded U.S. forces in the Gulf of Sidra and says Reagan's extravagant denunciations of Qaddafi have "made a hero of a minor eccentric" in the Arab world and may soon make a martyr of him.

Reagan understands that rhetoric has consequences. When he wanted to make a show of improving U.S.-Soviet relations, he dropped primitive references to the Soviet Union as "the focus of evil." At last week's news conference, he gracefully turned aside criticisms by Soviet leader Mikhail Gorbachev, whom Reagan said was still dealing in "the spirit of Geneva." On the same day, Reagan also gave soft answers to hard questions about South African President P.W. Botha and former Philippine president Ferdinand Marcos. He was pejorative only in reference to Qaddafi.

The standard explanation for Reagan's anti-Qaddafi rhetoric is that he has been simultaneously trying to provoke the Libyan leader while preparing the American public for military action. This seemed to be the unstated purpose of Prairie Fire, the massive naval exercise that produced an anticipated Libyan response and an easy U.S. "victory."

The doubts of some U.S. military planners that a naval armada was needed to demonstrate the recognized fact that the Gulf of Sidra is international waters were submerged by the desire to "do something" about Qaddafi. What might have been an effective limited demonstration of navigation rights instead became a deadly carnival advertised as an exercise of counterterrorism.

Those familiar with Reagan's feelings are inclined to think that his rhetoric reflects frustration as much as calculation. The president prides himself on making good on his promises, and is aware that he has failed to keep his pledge that terrorist acts against Americans would be met with "swift and effective retribution."

When Reagan uttered these words to the U.S. hostages returning from Iran a week after he become president, he did not understand the difficulties of building a coherent antiterrorist policy. He did not realize the depth of leftover skepticism pervading the U.S. military establishment after Vietnam about undefined military involvement on behalf of diplomatic goals. He did not account for the different diplomatic sensibilities, not to mention greed and fear, of allies that have important economic relations with nations harboring terrorists. He did not understand that focusing world notoriety on Arab extremists tends to enhance their prestige.

Overall, in dealing with terrorism, Reagan has been guided by a series of impulses posing as a policy. One is an understandable American impulse to take direct revenge for barbarous crimes against innocents. Another is the decent impulse of sparing other innocents in the process of retaliation. Still another is the desire to demonstrate that the United States is no longer the hapless giant that Reagan believed it to be during the Iranian hostage crisis.

These impulses explain but do not excuse a rhetorical descent to Qaddafi's level. While Reagan's frustration is understandable and widely shared, it is not a substitute for calm and reasoned policy designed to isolate Qaddafi.

Reaganism of the Week: Asked last Wednesday at the American Society of Newspaper Editors convention whether Marcos had spoken "a falsehood" in saying that he was unaware he was being taken out of the Philippines when he left office, the president replied: "I think maybe he was misinformed. It happens to every president."