Professing dissatisfaction with the U.S.- Israeli formula for a Geneva peace conference, the Palestine Liberation Organization appears to have hardened its position on the issue of Palestinian representation at the Middle East talks.

Initial hints of flexibility from the Palestinians and their supporters in Syria have given way to renewed militancy.

Secretary of State Cyrus Vance said last week that the question of Palestinian representation was "for the parties to decide." But the Israeli and Palestinian positions appear so far apart as to be irreconcilable. If Syria backs the PLO, as seems likely, the Geneva conference could be in jeopardy.

It is difficult to determine whether fresh PLO militancy is calculated to put pressure on Arab governments or whether it reflects frustrations and internal pressures by more radical Palestinian elements.

Procedural devices were advanced during recent talks in New York to get around Israel's objections to inviting the PLO to Geneva. According to Arab sources, an invitation could be extended to the Arab League which in turn would invite Egypt, Jordan, Syria, Lebanon and Palestinian to go to Geneva in the context of a single Arab delegation, a notion Israel had accepted.

Just who would comprise the Palestinian group is the crucial problem, however.

The PLO, at this juncture in the negotiating process, has apparently persuaded other Palestinians not to go along with attempts to find alternative representatives to replace the PLO at Geneva - as the Israelis and Americans have suggested.

Prominent Palestinian leaders on the occupied West Bank of the Jordan River have said they would not take such a role unless the PLO leaders request it, and the Jordanians are reported to feel the same way.

Any Palestinian who went to Geneva claiming to represent the Palestinian people without PLO sanction would be unacceptable to most of the Arab world. In 1974, the Arab designated the PLO as the sole legitimate representative of the Palestinian people.

After a meeting in Damascus last week between Syrian President Hafez Assad and Abu Iyyad, one of the dominant PLO figures. Damascus Radio reported the Syria recognizes "the right of the PLO as the sole legitimate representation of the Palestinian people and the right of his people to repatriation, self-determination and the establishment of an independent state of it own."

Palestinian sources here say they have been pleased with a new firmness in Syrian support as it has become clear that Israel is not prepared to deal with PLO.

Syrian Foreign Minister Abdel Halim Khaddam was quoted last week as saying his country would "never go to Geneva without the PLO."

The PLO's current position was summed up by its political affairs spokesman, Farouk Kaddoumi, in his speech at the U.N. General Assembly - no peace without the Palestinians, no Palestinians without the PLO.

The PLO executive and central committees are planning to meet soon to reaffirm that, observers here say, Any public suggestion that the PLO might settle for less has been dropped in the face of Israel's adamant posture.

The U.S.-Israeli working paper speaks of an unified Arab delegation that would include Palestinian Arabs, but Foreign Minister Moshe Dayan has made it clear that this does not mean anyone who has any PLO connections or anyone who lives outside the territories occupied by Israel in 1967, as most Palestinian leaders do.

Palestine National Council chairman Khaled Fahoum, who lives in Damascus and has close ties to the Syrians, called the working paper "totally unacceptable."

Abdul Mohsin Abu Maizer, the official PLO spokesman, said in Paris: "We reject this document because it tries to liqudate the question of national rights for the Palestinian people and opens the way for bulateral solutions." This way apparently a reference to the Syrian and Palestinian fear that Egypt, in its anxiety for a settlement with Israel, would go to Geneva on the basis of the working paper and negotiate separately with the Israelis, as it did after the 1973 war.

PLO officials emphasize that far from refusing to go to Geneva, which would imply recognition of Israel, they are anxious to do so, provided they get something in return.

Mahmoud Labadi, a PLO spokesman here, said the Americans were asking the PLO to recognize Israel "just so they will talk to us. That's not good enough. So we refuse and the Syrians refuse."

Another Palestinian source said that "in some ways the matter of recognition of the PLO at Geneva is more fundamental to us than the results of the conference . . . in negotiations, if it comes to recognizing Israel in exchange for getting our own state, all right, we can explain that to our people. But if we don't even get invited, how can we explain that?

"You have to understand that for ten years the Palestinian movement has failed in everything it has tried to do. We did not liberate the land, we did not eliminate Israel, we did not revolutionize the Arab world, we did not get our people out of the camps. The only things we have achieved is get the world to recognize us, to acknowledge our existence and accept our organization. If we don't go to Geneva we lose even that. How can we accept thatafter ten years of struggle?"