Libyan leader Muammar Qaddafi, stepping up his challenge to the Reagan administration, declared publicly tonight that he will train and equip Arab guerrillas for "terrorist and suicide missions."

Libya, he said, is "a base for the liberation of Palestine," and, he added, if Israel or the United States attacked it, he would "use all forms of violence in self-defense."

Although Qaddafi spoke mainly of the "unconventional" weapons at his disposal, his remarks came a day after Soviet warships arrived in Tripoli harbor. A fleet of U.S. warships, meanwhile, is deploying off the Libyan coast.

Appearing tonight for two hours in the hall of the National People's Congress before a crowd of about 2,000 madly cheering, olive-uniformed high school and college students and a handful of radical Arab figures including dissident Palestinian leader Abu Musa, Qaddafi abandoned completely the conciliatory tone toward the United States that he had used in interviews with western reporters during the past week.

In the face of repeated charges by the Reagan administration that Qaddafi already is training and supplying terrorists, including the group headed by Palestinian radical Abu Nidal that is blamed for the Dec. 27 attacks on the airports in Rome and Vienna, Qaddafi had taken pains until now to deny direct complicity with these groups or their actions.

But for much of this month, Arab People's Congresses have been meeting here. These groups of radical Tunisians, Egyptians and Palestinians, among others, passed resolutions vowing to form suicide squads and do whatever is necessary to free Palestine from what is called Israel's "Zionist" occupation.

Last night Qaddafi had chided these groups, openly wondering whether they would act on their motion.

Tonight he took the occasion of the anniversary of the birth of his idol, Egyptian president Gamal Abdel Nasser, to declare that "these Arab congresses should be in permanent session" here.

"I accept all their resolutions," Qaddafi said. "I announce that they will be trained for terrorist and suicide missions. We shall allocate trainers to train them and place all weapons needed to perform these missions at their disposal."

Qaddafi said he would offer, to the best of his ability, personal protection "to these volunteers, with the Palestinians at their vanguard," because "Libya is a base for the liberation of Palestine."

He told Western European diplomats and a group of reporters last week that he would encourage Palestinian guerrilla groups to confine their attacks to Israel and the occupied territories.

But one of Abu Musa's top aides in the audience tonight said that the Palestinians here had rejected this advice. "Qaddafi can say what he thinks, but the Palestinians will do what they like," he said.

Palestinians present included mainly guerrilla leaders opposed to Palestine Liberation Organization Chairman Yasser Arafat. In addition to Abu Musa and his aide, Abu Ali, who led a 1983 rebellion within Arafat's Fatah faction, there was also Abu Farras, of the dissident Popular Front for the Liberation of Palestine-General Command. Abu Nidal apparently was not there.

Tonight Qaddafi said, in an evident reference to Israel's Oct. 1 raid on PLO headquarters in Tunisia, "If the Israelis pursue the Palestinians outside of Palestine, it is the right of the Palestinians to pursue the Israelis anywhere in the world. The Rome and Vienna attacks targeted the Israeli airline counters."

"America's fleet, its threats and economic warfare -- neither this nor that will make us give up our support for the Palestinian cause or keep us from supporting it with everything at our disposal, because the Palestinians are fighting for a holy cause," Qaddafi said.

His remarks came as more U.S. and Soviet warships were gathering in the Mediterranean. Pentagon officials said a U.S. task force -- the aircraft carrier Saratoga, a guided-missile cruiser, a destroyer and an ammunition ship -- were steaming through the Suez Canal for the Mediterranean, where another carrier group headed by the USS Coral Sea is already deployed.

A Soviet guided-missile cruiser and frigate passed through the Bosporus strait early today and they will bring to 28 the number of Soviet vessels in the Mediterranean. The Soviets have stationed the flagship of their Mediterranean fleet, a submarine tender packed with electronic gear, in Tripoli harbor.

Qaddafi, readying his country for a possible violent U.S. or Israeli reaction, repeated a call for volunteer soldiers from all over the Arab world to join his armed forces.

"Libya is a base for confrontation expressing the pride and the nationalism of the Arabs," he said.

But he appeared to put his main hope for defense in unconventional forms of warfare, acknowledging that his regular armed forces would be a poor match for the full might of the United States.

Qaddafi cited the plot of "Fifth Horseman," a 1981 novel about him by Larry Collins and Dominique LaPierre, as a "symbol" of the power represented by the guerrillas he supports. In the novel, Qaddafi provides Palestinian terrorists with a nuclear device that they nearly succeed in exploding in New York City.

Qaddafi dismissed as impossible the nuclear aspects of the plot but he said it represented the fear America has of the kinds of fighters he uses