The difficulties apparent in this week's Jordanian-Palestinian talks here suggest that the United States faces a protracted period of tortuous diplomacy if President Reagan's Middle East peace plan is to succeed.

King Hussein failed to move Palestine Liberation Organization Chairman Yasser Arafat beyond saying in public that he would consider some future link between Jordan and a Palestinian entity as envisaged by Reagan. U.S. diplomats had hoped that Arafat would give the Jordanian monarch a clear and public mandate to negotiate for the Arabs in the context of the Reagan plan.

Arafat did less than that. He did not give Hussein any solid new concession from the PLO that Reagan could show Israel in pressing for flexibility there. The American president has said the United States will not deal with the PLO until Arafat openly accepts Israel's right to exist.

But the PLO chief did keep the U.S. plan alive by tacitly giving Hussein the right to mediate between Washington and the PLO on that issue. Arafat pointedly referred to "positive elements" in the Reagan plan, and said that the Palestinians were prepared to keep an open mind on an ultimate confederation with Jordan, an arrangement that seems close to what Reagan has proposed. However the PLO demands establishment of its own state prior to any federation.

Given Reagan's goal of achieving a comprehensive peace settlement in the region by providing self-government for the Palestinians in land now occupied by Israel, all signs suggest that if progress comes it will be in small steps and require lengthy bargaining with a variety of parties.

PLO officials suggested that they considered the implicit recognition of Israel in the Fez summit resolutions a major concession and that they were not willing to go further. The officials said that the importance of the Fez conference had not been appreciated abroad, indicating that they thought that one major concession already had been made.

"It is very hard to build and very easy to refuse or destroy," a Jordanian official said in commenting on the talks here.

The talks brought together two leaders with many years of ties to the Arab residents of the West Bank. Most people outside Israel -- and some inside it -- view either Hussein or Arafat as the most legitimate spokesman for the Palestinians in the occupied territories. Hussein reigned over the West Bank before Israel seized it in the 1967 war, and many polls have shown that the PLO enjoys support among residents there.

Prospects for Reagan's plan thus depend greatly on whether Hussein and Arafat can work together in trying to define a future for the occupied territories -- including the Gaza Strip,controlled by Egypt before 1967 but generally expected to be part of any future Palestinian entity.

The centerpiece of the Reagan plan is that a self-governing Palestinian entity "associated" with Jordan should be created in territories now occupied by Israel. The proposal is similar to Hussein's call for a Jordanian-Palestinian federation.

Reagan explicitly rejected an independent Palestinian state, long the PLO's goal. Israel refuses to consider establishment of such a state on grounds that it would threaten Israeli security.

A central reason for the importance of the talks is that Hussein is not considered likely to break with the PLO and make a separate peace with Israel as did Egypt's Anwar Sadat, according to Arab and Western sources here. The king is a cautious, pragmatic leader and generally shuns dramatic gestures of the type associated with Sadat.

Jordan is too small and too poor to be likely to run the risk of being ostracized by the Arab world if it tried to deal on its own on behalf of the Palestinians.

Hussein would like to have authority to handle the Palestinian issue outside the pan-Arab context, the sources said. This also would please the United States, which needs somebody acceptable to the Israelis to represent the Palestinians.

But Arab nations are unwilling to see a single country in the vanguard of the Palestinian cause, and the Fez summit last month upheld the PLO as the Palestinians' sole voice.

Syria suggested during the talks that Arafat would face a split in the PLO if he went ahead with his rapprochement with Jordan, Damascus' longstanding rival. There also are signs that Saudi Arabia and Kuwait are unhappy about emergence of a Jordanian-Palestinian axis; they have trimmed back their financial contributions to Jordan, leading Hussein to complain recently that the country was short $361 million of expected aid.

Nevertheless PLO and Jordanian sources confirmed that Arafat effectively had given Hussein a green light to serve as an informal mediator with the United States.

Arafat and the PLO have been thrown into Hussein's lap to some extent because of their eviction from Beirut. Jordan is the Arab country bordering on the territory that PLO desires as a homeland, and one diplomat here said the PLO's alternative to dealing with Hussein was to become a network of guerrillas scattered around the world like the Armenian terrorists.

PLO officials stressed that one of their most important concerns was confirming that Reagan was committed to trying to put the plan in effect -- which means forcing Israel to withdraw from the occupied lands.

"Let us pray that Reagan is strong enough to implement this plan," PLO spokesman Mahmoud Labadi said. But Arafat's ability to make concessions was limited by the pressure from Syria and by his own longstanding demand for a state and not just a federation with Jordan.

One private observer here suggests that Arafat may be pursuing an even more conspiratorial plan of allowing Hussein to associate himself with the Reagan plan and become discredited should the Palestine National Council refuse to accept it.

Arafat might then hope to replace Hussein as head of Jordan, using Syrian help to make Jordan a Palestinian state -- as the Israelis insist it already is, because more than half of the population is Palestinian.

The Jordanian newspaper Al Dustour said one major goal of the talks was to "narrow the maneuverability" of Israeli diplomacy and thus keep the Jewish state on the defensive. They appeared to have succeeded in that mission by not closing the door on the Reagan plan, but the U.S. diplomatic work on the initiative apparently is just beginning.