The Reagan administration, expressing anger at Italy's "incomprehensible" release of a Palestinian leader accused of masterminding the hijacking of an Italian cruise ship, said yesterday it anticipates that Yugoslavia also will reject U.S. demands for the arrest of Mohammed Abbas.

In Belgrade, Yugoslav officials made no official comment about the status of Abbas, who arrived there Saturday night after Italian authorities spirited him aboard a Yugoslav airliner at an airport near Rome.

But FBI Director William H. Webster said Yugoslavia has "declined to detain" Abbas, who was briefly in Italian custody after he and four accused hijackers were caught aboard an Egyptian airliner that was diverted to Sicily by U.S. Navy jets on Thursday.

U.S. officials, noting Yugoslavia's close ties with the Palestine Liberation Organization, acknowledged that they have little hope that Yugoslavia will hold him until the United States can request his extradition.

Reports from Belgrade quoted a Yugoslav government spokesman, who declined to be identified, as saying, "We have good relations with the PLO, and we support the Palestinian cause. You can draw your own conclusions."

That indicated that the United States probably will be frustrated in its efforts to prosecute Abbas, whom it has called "one of the most notorious Palestinian terrorists," for his alleged role in planning and controlling the hijacking of the cruise ship Achille Lauro last Monday. An American citizen was shot to death before the four hijackers surrendered to Egyptian authorities last Thursday.

The administration's determined pursuit of the hijackers has brought the United States into actual or potential conflict with three countries -- Egypt, Italy and Yugoslavia -- since Thursday night when U.S. Navy fighters intercepted the Egyptian jetliner and forced it to land in Sicily. In addition to the four hijackers, the Egyptian plane was carrying Abbas and another unidentified Palestinian, who the Egyptian government said had acted as mediators in arranging the surrender of the four. The whereabouts of the unidentified Palestinian was not known.

The U.S. move against the plane triggered violent, anti-American demonstrations by Egyptian students and caused President Hosni Mubarak to complain bitterly about the incident. In an attempt to ease the growing tension between Washington and Cairo, the U.S. ambassador in Egypt, Nicholas Veliotes, summoned reporters yesterday to praise Egypt's role in freeing the Achille Lauro, and he noted: "We have no interest in offending the government or people of Egypt. We are friends."

Italian Prime Minister Bettino Craxi's government has said it will prosecute the four for piracy and hostage-taking. But, although the administration obtained an arrest warrant for Abbas from the U.S. District Court here and rushed it to Rome, the Craxi government released Abbas on the grounds that Italian judicial authorities did not consider the U.S. evidence sufficient to justify his continued detention. Craxi has tried to cultivate good relations with the PLO and Arab states.

In the initial U.S. public reaction Saturday night, the State Department, while expressing disappointment at Italy's action, also stressed America's "very broad and deep relationship with this valued NATO ally." Early yesterday, however, White House spokesman Larry Speakes issued a revised statement that eliminated the praise for the U.S.-Italian relationship. It also called the Italian decision "incomprehensible" and concluded:

"The United States government is astonished at this breach of any reasonable standard of due process and is deeply disappointed."

White House officials, while saying they did not want a serious, long-term breach in U.S.-Italian relations, acknowledged that the references to friendship had been deleted because President Reagan wanted to underscore to the Italians his disappointment with the release of Abbas and the manner in which it was done.

That tone of sharp reproach also was evident yesterday in the actions and statements of other senior U.S. officials. In Rome, U.S. Ambassador Maxwell M. Rabb met for 2 1/2 hours with Italian Foreign Minister Giulio Andreotti to express Washington's displeasure, and Treasury Secretary James A. Baker III, in an apparent reference to Italy's policy of friendship for the PLO, questioned "whether the release was purely on judicial and legal grounds or whether or not somehow politics might have been involved."

"I don't know what the political reasons are," Baker said on NBC News' "Meet the Press." "I'm simply saying that we felt our evidence was extremely good evidence. We supplied it to the Italian government. Nevertheless, in the face of what we thought was rather compelling evidence, the release was made within hours."

Abraham D. Sofaer, the State Department's legal adviser, interviewed on ABC News' "This Week With David Brinkley," said that the evidence turned over both to Italy and Yugoslavia included "transcripts." He did not elaborate, and other U.S. officials refused to discuss details on the evidence on the grounds that it involved material obtained through intelligence means.

It is believed that the United States, possibly with help from Israel, was able to monitor radio communications between the PLO and the hijackers aboard the Achille Lauro. The transcripts of these conversations are understood to form the basis of the U.S. charges against Abbas, who also is known as Abu Khaled and who heads a faction of the Palestine Liberation Front guerrilla group closely allied with PLO Chairman Yasser Arafat.

Webster, who appeared on CBS News' "Face the Nation," said that "the Yugoslav authorities have declined to detain Abbas under the request we made. So I anticipate he probably will move along. I understand he is operating under an Iraqi passport, and we have reason to believe he is in the PLO embassy in Belgrade . . . . "

Webster said "prudent precautions" are being taken against possible Palestinian reprisals. But he added that the danger of an attack inside the United States has "probably been a little overstated," and, he continued, "Most of the rhetoric promising retaliation has come from outside the United States. I don't think it's going to happen."

Reuter news agency reported that when Abbas arrived in the Yugoslav resort city of Dubrovnik en route to Belgrade Saturday night, he vowed to free the four hijackers held in Italy. Reuter also said he accused the United States of state terrorism and quoted him as saying: "We the Palestinians use handguns in our liberation struggle. They use fighter planes and aircraft carriers."