Libyan leader Col. Muammar Qaddafi denied yesterday that he has sent agents to assassinate high-ranking U.S. officials, but the State Department declared it has "strong evidence" of such a plot.

Qaddafi's denial via satellite to an American television program followed an increase in security for President Reagan and several of his senior Cabinet members and White House aides because of reports of a Libyan-generated assassination conspiracy.

The Libyan leader, expressing irritation at widespread reports of a plot, charged that he is the target rather than the perpetrator of assassination efforts.

"We refuse to assassinate any person . . . . It is not our character, not our behavior, to assassinate any person. It is the behavior of America, preparing to assassinate me, to poison my food. They tried many things to do this," Qaddafi said.

Speaking on "This Week With David Brinkley" (ABC, WJLA), Qaddafi also called Reagan "silly" and "ignorant" to believe the assassination reports and "a liar" to spread them. Reagan's behavior is "like a child," rather than that fitting for the leadership of a great power, he charged.

State Department spokesman Susan Pittman, in a statement released several hours later, said, "We have strong evidence that Qaddafi has been plotting the murder of American officials both here and overseas."

She added, "We certainly hope that Qaddafi's denial means that he will abandon the use of terrorism and assassination as a part of his foreign policy. When he has stopped, we will know it."

The globe-girdling argument with Qaddafi came as the administration prepared for White House decision-making on economic measures against Libya that have been under consideration for many weeks.

Awaiting presidential decisions at a meeting of the National Security Council, according to official sources, are such measures as a unilateral U.S. boycott of Libyan oil and a renewed and strengthened call for evacuation of American citizens from that country. Reagan has opposed a unilateral boycott, but political backing for such a measure has been growing.

Almost 2,000 Americans, mostly oil company workers and their dependents, remain in Libya. The Americans' presence has been a limiting factor on Washington policymaking because of concern that they might be harmed or taken hostage in retaliation for strong U.S. measures.

Sen. Daniel Patrick Moynihan (D-N.Y.), acting chairman of the Senate Intelligence Committee, interviewed on the same program, said "there will be the most severe consequences" if any violence is done in a Libyan-backed conspiracy to a U.S. official.

He refused to elaborate on possible U.S. responses but said, "They Libyans should know it's going to be extremely unpleasant."

Moynihan, who estimated the reliability of the intelligence reports as a "point eight probability," evidently meaning an 80 percent probability, said the U.S. tracking of Qaddafi's terrorist activities dates to 1977.

In early 1977, according to authoritative sources, the Carter administration was informed that Qaddafi had assigned a "hit team" to assassinate U.S. Ambassador to Egypt Hermann Eilts.

President Carter is reported to have sent a strong protest to Qaddafi and, after the Libyan disputed the charge, supplied a detailed account of what the U.S. government knew about the plot.

Following the shooting down of two Libyan aircraft by U.S. warplanes in the Gulf of Sidra last August, there were reports that Qaddafi had ordered the assassination of U.S. diplomats in Europe in retaliation. Some U.S. officials blamed Libya for the Nov. 12 assassination attempt on Christian Chapman, U.S. charge d'affaires in Paris, but no evidence was produced.

Senate Democratic leader Robert C. Byrd (W. Va.) said yesterday that he had been briefed by the Secret Service on the Libyan assassination reports.

"In my experience here in Washington, I haven't seen anything like this . . . . I don't think that we can take this evidence lightly. I think it is in the interests of ourselves that we take it seriously and that we take every precaution," he said on "Face the Nation" (CBS, WDVM.)

Tighter security also has been ordered for Vice President Bush, Secretary of State Alexander M. Haig Jr., Secretary of Defense Caspar W. Weinberger, White House chief of staff James A. Baker III, counselor Edwin Meese III and deputy chief of staff Michael K. Deaver.

Qaddafi, asked in yesterday's interview about possible revenge for the shooting down of his planes, responded: "It is is not a matter of revenge, but it is a matter of resistance against American aggression . . . . We are facing this aggression again, and we are not afraid."

He called on the American people to get rid of Reagan and "this bad administration" as they did with President Nixon who resigned in 1974 in the wake of Watergate revelations. Qaddafi said he continues to favor good relations with the United States.

Qaddafi said he has taken "all the preparations" in case American oil corporations withdraw from Libya. Exxon announced last month that it is halting its operations there and withdrawing all of its personnel, while Mobil has said it is studying its future in Libya.

Other U.S. companies operating there are Occidental Petroleum, Marathon, Conoco and Amerada-Hess.