President Reagan last night imposed a total U.S. economic boycott against Libya in response to the terrorist attacks in Rome and Vienna last month and condemned Libyan leader Muammar Qaddafi as a "barbarian" who should be punished for the deaths of 19 people, including five Americans, killed in those raids.

In his first nationally televised news conference in nearly four months, Reagan showered the Libyan leader with denunciations for "criminal outrages by an outlaw regime" and described him as "flaky" while vowing that "further steps will be taken" by the United States if the economic sanctions "do not end Qaddafi's terrorism."

"Qaddafi deserves to be treated as a pariah in the world community," he said.

Reagan announced he has signed an executive order seeking to end direct trade and economic activities between the United States and Libya. He also called for the 1,000 to 1,500 Americans working in Libya to return to the United States. His order would impose criminal penalties on companies or individuals who defy it.

Reagan demanded that Qaddafi not harm the Americans in Libya as they attempt to leave.

The president acknowledged repeatedly during the 35-minute news conference that he confronts limits on how far the United States can go in fulfilling his 1981 promise of "swift and effective retribution" for terrorist attacks such as the Dec. 27 shootings at the Rome and Vienna airports. Reagan held open the possibility of U.S. military action but refrained from making direct threats about using it.

Also last night, Reagan defended a recent secret directive that expanded the use of polygraph, or lie detector, tests for thousands of government employes who have access to classified information. Reagan denied that he had changed his mind about the order after protests from Secretary of State George P. Shultz or that he had signed it without being aware of the scope of the polygraph examinations it would require.

Questioned about the balanced-budget legislation he endorsed and recently signed, Reagan stood fast for his goals of a 3 percent increase in defense spending next year above inflation and no tax increase. Reagan complained that "I got burned" by agreeing to a tax increase in 1982, because Congress did not deliver the promised spending cuts to go with it.

While many members of Congress and some administration officials have said a tax increase this year may be inevitable, Reagan said he would not consider one until a stronger effort is made to reduce the deficit through spending cuts.

Responding to a question about the timing of Soviet leader Mikhail Gorbachev's planned visit to the United States this year for a second summit conference, the president said he wanted to meet in June but that the Soviets have suggested autumn, without giving an explanation. Others have suggested that a later meeting could give the superpowers more time to work out an initial accord on reducing nuclear weapons.

Reagan last night held the 33rd news conference of his presidency. His predecessor, Jimmy Carter, held 59 such sessions during his 48 months in office.

Reagan stumbled verbally a few times last night in response to questions. He made an incorrect statement about the impact of his tax cuts on government revenues, cited the wrong U.N. resolutions pertaining to Israel, said "Vietnam" when he meant Vienna and confused Gorbachev's title.

But Reagan also responded with some one-liners. About Qaddafi, he said, "I find he's not only a barbarian, but he's flaky."

In his opening statement, Reagan said responsibility for the terrorist attacks "lies squarely with the terrorist known as Abu Nidal and his organization" but added that the shootings could not have taken place "without the sanctuary and support provided by regimes such as Col. Qaddafi's in Libya."

While refusing to provide details, Reagan said the United States has "irrefutable evidence" of Libya's role in the attacks, including the location of terrorist training camps. Reagan said that even if Qaddafi personally was not involved, his government was, "and I don't think his regime is doing anything without his guidance."

"I don't think he's capable of telling the truth about these things," Reagan said of Qaddafi. "But we do know . . . the location of training camps for terrorists, and we also know that Abu Nidal has more or less moved his headquarters into Libya. So we speak with confidence."

By providing aid to terrorist groups, Reagan said, "Libya has engaged in armed aggression against the United States under established principles of international law, just as if he had used its own armed forces."

The president said the economic boycott against Libya, which tightens the sanctions he imposed against Qaddafi in his first term, is justified on grounds that "terrorists, and those who harbor them, must be denied sympathy, safe haven and support."

Reagan said the measures he was putting into effect would "impose a total ban on direct imports and export trade with Libya, except for humanitarian purposes. They prohibit commercial contacts and other transactions with Libya, including travel-related activities other than those needed for journalism . . . .

"Civilized nations cannot continue to tolerate, in the name of material gain and self-interest, the murder of innocents," Reagan said. Despite rebuffs from Western European allies to his earlier requests that they impose economic sanctions on Libya, Reagan said he would continue to try to persuade them to isolate Qaddafi.

"Americans will not understand other nations moving into Libya to take commerical advantage of our departure," he said, pointedly quoting a statement by Italian Prime Minister Bettino Craxi on the need for taking action against nations that provide training and arms to terrorists. Italy is Libya's largest trading partner.

Reagan acknowledged once again last night the difficulties of military retaliation against such attacks.

Referring to what he called "all this talk that there has been about harsh talk and no action," Reagan recalled the suicide bombing attacks on the U.S. Embassy and the Marine headquarters in Lebanon. In both, he said, "the perpetrators of those acts died with the victims."

"Now, we have made every effort to try and establish, well, who brought these people there? They certainly can't be questioned. How did they get there? Now we have had two more recent attacks. But in these two attacks the perpetrators are either dead -- killed in the scene -- or they are wounded and in hospitals under arrest.

"The only actual case where there were terrorists, and there they were and we knew their location and where they were trying to go and we brought them down -- and that was the Achille Lauro case . . . , " he said.

"I know it appears that we sit here and are not doing anything," the president said. He then claimed, without providing details, that the United States has "actually recorded in the last year, and know, that we have aborted 126 terrorist missions." He indicated that some were in the United States.

Reagan, who has ruled out the use of American military force for retaliation if innocent people are endangered, last night said the presence of the Americans in Libya was a consideration in responding to the recent terrorist attacks. Those Americans are "potential hostages," he said.

The president said the departure of the American carrier Coral Sea from port in Naples, Italy, was routine, but other U.S. officials have said the escalated presence of U.S. forces in the region is designed to keep Qaddafi unsettled about a possible military strike.

"I am not going to make any comment as to whether we have other actions in mind or what might be done," Reagan said. "I think that Mr. Qaddafi would be very happy if I did answer such a question, but don't -- I'm not interested in making him happy."

Reagan also acknowledged that the Western Europeans have not cooperated so far. "I don't know that we're going to outright ask them" to join the U.S. sanctions, he said, but rather "we're going to tell them what we're doing" and point out that the latest sanctions are not as "ambiguous" as earlier measures.

He conceded that some of the allies may have strong economic ties to Libya "that's just going to render this nearly impossible . . . . It may be frustrating, but we're going to go on with what we think has to be done."

Asked whether the actions against Libya will affect the American hostages in Lebanon, Reagan said, "We are talking about two separate subjects here" and promised that the United States will continue seeking their release. Reagan said the missing Americans are "in as reasonably good health as could be expected in view of their incarceration."