The futile debate at the United Nations shows that the Carter administration's latest effort to engage the Palestine Liberation Organization never had a chance. But why did the president and all his advisers -- including Robert Strauss -- embark upon a policy foredoomed to failure? And what is the cost?

The reason the initiative could not succeed lies in the basic political scene in the Middle East. That scene is now dominated by the Egyptian -- Israeli peace treaty, which has set in motion an ongoing process of political, diplomatic and commercial ties.

To be sure, progress in one area -- achieving autonomy for the Palestinian Arabs -- has been slow. The Israelis and the Egyptians have not found it easy to define autonomy. The local Arabs, in keeping with the contention of th PLO that the treaty sells out, their rights, have boycotted the negotiations.

Still the talks continue. Some Palestinians might eventually join. An offer to territory on the West Bank might eventually prompt King Hussein of Jordan to enter the negotiations as surrogate for the Palestinians. These possibilities depend less on Israel and Egypt than upon the staying power of the Arab groups banded together against the peace treaty in the "Rejectionist Front."

The Rejectionist Front has its cutting edge in the PLO. But the base of PLO strength is the connection with Syria and Iraq.

Apart from providing shelter, finance, arms and access to Moscow and other radical Arab states, the radical regimes in Damascus and Baghdad back up PLO threats to make trouble for conservative regimes in the Persian Gulf, especially Saudis joined the rejectionists in sanctions aimed against Egyptian follow-through in the treaty with Israel.

In the past few months, however, the Baghdad connection has started to unravel. The inability of the Ayatollah Komeini to stabilize Iran has brought troubles to neighboring Iraq. The Iraqui leader, Saddam Hussein, has instituted a kind of coup that poisons relations with Syria.

The PLO, with the sands shifting against it, was in no position to take a moderate stance regarding Israel. Thus on Aug. 10 the PLO Central Commission met in Damascus and voted against any U.N. resolution that did not acknowledge the right to a Palestinian state.

The tough stand of the PLO obliged the Israelis to resist the American effort. The Egyptians also had to refuse the gambit. Hence the United States found itself backing a maneuver rejected by the PLO and both Egypt and Israel.

Part of the reason, no doubt, is that the president, Secretary of State Vance, the national security adviser, Zbigniew Brzezinski and Ambassador Andrew Young have always believed in bringing the PLO into the diplomatic process. In addition, there has been a misreading of conditions in Saudi Arabia.

Early in July, as an Independence Day gift to President Carter, the Saudis raised oil exports from 8.5 to 9.5 million barrels a day. That move was accompanied by Saudi urging that Washington make accommodations with the PLO.

A realistic reading of that exhortation is that the Saudis wanted credit in Syria and Iraq and with the PLO for Carter administration, including Ambassador Strauss, took the Saudis at face value. As a means of currying favor in Riyadh, they knocked themselves out to be nice to the PLO.

As a result of that error, the cost of the abortive initiative is apt to be very high. The PLO has been rewarded for a show of intransigence. The Rejectionist Front has a new incentive to continue its challenge to Israel and Egypt. The Saudis will be more than ever obliged to wield the oil weapon on behalf of the PLO.

Other Palestinians are even less likely to enter the autonomy talks between Israel and Egypt. King Hussein, far from joining the talks, is now apt to cut a deal with the PLO.

Even Mr. Carter cannot charge these costs to the account of the Washington press corps. In fact, the blame attaches to Carter's singular presidency. The strong sense of self-righteousness and weak sence of national interest at first baffled the world. Now Mr. Carter has been found out, and the country is beginning to pay for his administration. And the country, far from continuing to show indulgence, needs to realize that until Carter is out of the White House, it will continue to pay and pay and pay.