A Carter administration effort to postpone the U.N. Security Council debate on the Middle East apparently failed today, setting the stage for a U.S. veto of a resolution supporting "self-determination" for the Palestinians.

Acting at the instruction of President Carter, U.S. Ambassador Andrew Young met through the afternoon with other members of the Security Council, trying to persuade them to accept a postponement. But that effort ended without success, as Arab sponsors of the pro-Palestinian resolution said they will press for a vote.

According to Robert S. Strauss, special presidential negotiator for the Middle East, the U.S. will probably veto the resolution if it comes to a vote -- a move that would undoubtedly anger Saudi Arabia, Kuwait and other moderate Arab states.

It was in a similar effort to postpone the same debate last month that Young held his now-famous meeting with a Palestine Liberation Organization observer, the action that led to his resignation under fire.

But, in the aftermath of his resignation -- with it apparent that the United States had no alternative to offer the Arab nations -- Young apparently failed today.

Kuwait's U.N. Ambassador Abdella Yaccoub Bishara left the consultations late this afternoon and told reporters that he will push to bring the Palestinian resolution to a vote and anticipates a U.S. veto.

Bishara has worked to moderate the conflict between all-out supporters of the PLO position and the United States. He played a key role in the July postponement, but the United States then was holding out the promise of a U.S.-introduced resolution that would have made Washington's position slightly more favorable toward Palestinian rights.

But the administration this week abandoned its effort to introduce the Palestinian resolution after Strauss got a chilly reception from both Israel and Egypt last weekend in the Middle East.

There will be a final round of secret consultations Thursday morning, and the Security Council debate is expected to begin in the afternoon.

Bishara, who arranged Young's July 26 meeting, with PLO observer Zehdi Labib Terzi, said there is no question in his mind that the debate will take place.

"It would take another Young-Terzi meeting to bring about a postponement," Bishara said. "If the Americans want a postponement, let Young talk to Terzi." Young has become a hero in the Arab world, the ambassador added.

Young, president of the Security Council by rotation, carried out the instructions of Washington seeking the postponement and aides said he would be in the chair at any Security Council meeting.

His aides said that if Washington orders a veto, Young will cast it despite his publicly proclaimed personal opposition to American policy procedures.

The Palestinian resolution that will be put before the council says, "That the Palestinian people should be enabled to exercise their inelienable rights of self-determination, national independence and sovereignty in Palestine."

It also reaffirms the long-standing U.S. resolution that affirm Israel's rights to exist and have defensible borders. In those earlier resolutions, however, the Palestinan issue is addressed only as a refugee problem.

The Security Council debate will come at what is perhaps a high-water mark of interest in the Palestinian cause in the United States and some other nations.

The resignation of Young under pressure has brought enormous media attention to the U.S. policy of having no contact with the PLO as long as it refuses to accept Israel's right to exist. Several nations are reevaluating their positions toward the PLO and the Palestinian's supporters probably have never been in such a powerful position at the United Nations.

The debate over which there has been so much preliminary wrangling will be a short one. Friday is a holiday for many Moslem nations because of the end of the holy month of Ramadan.

On Saturday, many of the key diplomats will be leaving the United Nations to begin preparations for the nonaligned meeting to be held in Havana next month.

Earlier today, President Carter made it clear that the U.S. goal at the United Nations was to seek a postponement of the pro-Palestinian resolution -- but he also made it clear that he was not confident of success.

"I don't know if the debate will be postponed, but that is what we would like," Carter said during a stop in Burlington, Iowa, during his Mississippi River vacation cruise aboard the Paddlewheeler Delta Queen.

Carter also sought to play down reports of dissension among his top makers of Middle East policy. He did not deny that there were disagreements, but charged that the press had "grossly exaggerated" the situation.

Reporters traveling with Carter's Mideast negotiator Strauss during his recent trip to Israel and Egypt, including The Washington Post's Edward Cody, qoted sources in the Strauss party as saying Strauss had not agreed with his instructions, which narrowly defined his negotiating position on a suggested U.S. compromise resolution on the Palestinian question. Both Israel and Egypt voiced strong objections to the proposal.

Returning to Washington, Strauss spoke to reporters on board his plane. New York Times reporter Bernard Gwertzman reported that Strauss intended to seek a clarification of who is in charge of American policy on the Middle East. The dispatch said Strauss was irritated at the strict instructions he was given and wanted a commitment from the president that he is in charge of Middle East policy, not Secretary of State Cyrus R. Vance or the national security adviser to the president, Zbigniew Brzezinski.

Apparently it was this report with which Carter took issue today. The president said:

"People out in the country are not nearly as excited as the press about personnel changes, or little transient squabbles that are exaggerated. There is no doubt in my mind the press grossly exaggerated what Bob Strauss said on his way back from the Middle East. I have talked to Bob and Cy Vance and others in the White House, Brzezinski, and others, and there's no serious thing there."

However, one person who was aboard the Strauss flight back to Washington Monday, and who heard the Strauss conversation involving The New York Times reporter, substantiated that newspaper's account. If anything, he said, the news article understated Strauss' comments.

Carter also said he had stage-managed, by long distance, the brief appearance in Washington Tuesday by Vance and Strauss, where Vance asserted that it was the secretary of state who is in charge of Mideast policy, and that Strauss is in charge of the peace negotiations. He said he had instructed that Vance and Strauss appear together before the press to show that they "are not enemies and we were not going to war within the administration while I was out of Washington."

Carter added that the decision to abandon the U.S. compromise resolution, seek a postponement of the debate, if that failed, and veto the Arab resolution, was a "recommendation that came from everyone unanimously."

Appearing on the NBC television show "Today," Strauss said that if the pro-Palestinian resolution were brought up in the Security Council by its Arab sponsors, "we will take such action as we deem appropriate and it will probably (be). . .a veto."

Strauss also sought to play down the previous reports of his disagreements with the earlier U.S. compromise effort and with his Carter administration colleagues.

He said everone in the administration had "common goals" but that "you don't get there in the same mode." He said: ". . .a lot of this is foolishness about strong disagreements. Cy Vance and I are probably the closest friends as you'll find in any government."

He added that the nature of his role and that of Vance "makes for a bit of awkwardness, of course, in the lines or responsibility."

Meanwhile, at the State Department, spokesman Tom Reston said the United States was "deeply concerned and saddened" by the renewed violence in southern Lebanon. Lebanese militia allied to Israel did most of the shelling, he said, but Palestinian forces did some shelling, too.