President Reagan, responding 11 days after terrorist attacks at the Rome and Vienna airports, is planning to announce tonight that the United States will impose further economic sanctions against Libya and again seek help from Western Europe in trying to isolate Libyan leader Muammar Qaddafi, administration sources said yesterday.

Sources also said Reagan intends to "keep his military muscle flexed" as part of the U.S. response to the attacks that killed 19 people, including five Americans. Reagan told aides yesterday he wants "to make Qaddafi go to sleep every night" unsure about possible U.S. military actions against Libya.

One senior official described the approach as "staying in position to keep 'em guessing" about the purpose of American naval units patrolling in the Mediterranean. Officials described the U.S. military maneuvers as mostly a tactic to threaten Qaddafi and said a strike against Libya now seemed unlikely.

Officials said Reagan intends to open his nationally televised news conference at 8 tonight with a statement of U.S. plans. Some officials said the sanctions would be "immediate and comprehensive" but did not disclose details. Others noted, however, that the United States has only limited economic leverage over Libya because trade is sharply reduced as a result of sanctions, still in effect, that were imposed in Reagan's first term. In addition, the effectiveness of U.S. sanctions is reduced as long as Western European allies continue to trade heavily with Tripoli.

While some allies, including West Germany, have rebuffed U.S. appeals to isolate Libya with an economic boycott, Reagan will renew his plea tonight while making another denunciation of Qaddafi's role in supporting terrorists such as the group responsible for the Dec. 27 attacks.

Reagan's decisions came as the rhetorical contest between the United States and Libya continued yesterday. Presidential spokesman Larry Speakes and State Department spokesman Charles Redman both said Sunday's denial from Qaddafi that there are terrorist training camps in Libya is "patently untrue." Speakes said the United States has "firm evidence" that the camps exist.

Sources said the administration was considering release of some aerial photographs, now classified, showing the training camps, in conjunction with Reagan's news conference statement. Speakes said separately yesterday that the evidence may not be released because it would expose intelligence-gathering methods.

The administration has accused Qaddafi of supplying training camps and money to terrorists such as Palestinian guerrilla Abu Nidal, who the administration has identified as the mastermind of the latest attacks.

But Speakes indicated that a military strike against Libya is unlikely, given the possible deaths of innocent people, which Reagan has said he wants to avoid, or reprisals against Americans working there.

"We want to be certain as to what we're doing and where we're striking," Speakes said. "We subscribe to norms of behavior that many other countries do not, particularly those that export terrorism, and we certainly would deal with what is acceptable to our own people as far as dealing with terrorism."

However, he also said Reagan "would like to be able to punish those responsible for these acts."

Qaddafi threatened last week and again Sunday to send suicide squads into the United States in response to any military attack. Speakes said yesterday that American authorities are "on guard" against the possibility.

On economic measures, Speakes said, "We would think that our European allies would be more likely than ever to join us in some kind of economic isolation of Libya."

Other officials said Reagan would point out that U.S. imports of Libyan oil and exports to Libya have fallen off sharply since 1981. "We're not asking people to do something we ourselves are not prepared to do," a White House official said in discussing Reagan's approach.

The official said the president was looking at "other measures" besides limiting exports to Libya, which totaled approximately $200 million in 1984, down from $860 million in 1979. One option under consideration would be to ask other nations to deny landing rights to Libyan commercial aircraft.