Under pressure from President Carter, the European allies are backing away from their plan to seek a U.N. Security Council resolution recognizing the rights of Palestinians to self-determination and their own homeland in the Middle East.

Last weekend, Carter threatened to use the U.S. veto in the Security Council to block such a resolution, and European diplomats now realize that it would be damaging to the Atlantic Alliance and peace prospects in the Middle East if they do not delay the move until after the U.S. presidential election, according to informed sources in Europe.

But the European allies are still determined to launch their own diplomatic initiative in the Middle East, these sources emphasized. Such an initiative would commit the Europeans to the principle of Palestinian self-determination and would seek a role for them in future Middle East peace negotiations.

The U.N. Security Council is expected to meet Wednesday to discuss developments in the Israel-occupied West Bank, and a special session of the General Assembly will be held next month to discuss the Palestinian question.

As an alternative to working through the United Nations, diplomats in Britain, France, West Germany and the other six nations of the European Economic Community are looking for ways to talk directly with Middle East leaders about how to involve the Palestinians in the peace process.

Informed sources suggested today that this could be done by opening separate European Community diplomatic channels to Israel and to the Tunis-based Arab League. These channels have been used only for technical trade discussions since Carter negotiated the Camp David accords between Egypt and Israel.

Last weekend, Carter warned the Europeans "not to intervene" in the stalled negotiations between Egypt and Israel on Palestinian autonomy in the Israeli-occupied West Bank and Gaza Strip. Carter said the United States serves in the "intermediary role" in these negotiations and warned that "any attempt that might be made by the European allies to circumvent or to replace this Camp David process would be a mistake."

This has angered European diplomats, who point out that the European allies have "legitimate interests" in the Middle East by virtue of historic ties and their dependence on Middle East oil and trade. European diplomats have also become convinced that resolving the Palestinian question is the key to future stability in the region.

The French government is reported by diplomatic sources to be the most angry and determined to demonstrate to the United States that the Europeans will push ahead with their own Middle East initiative. "Carter should have taken the French reaction into consideration before speaking so unsubtly," one source said.

British sources said it would now be "counterproductive" to seek a U.N. Security Council resolution or the plan that Britain had suggested as the foundation for a European initiative. "There is no reason to go ahead like a bull in a china shop," said a source here.

But a formal declaration by the European Community leaders at their summit later this month in Venice could be used to "codify" European recognition of Palestinian self-determination and the need for the Palestinians' participation in peace talks in exchange for Palestinian recognition of Israel's right to exist within secure borders.

"There also will have to be some real action to replace a U.N. resolution," said one source. "The Europeans want to be seen to be doing something. They are looking for some way to start talking to the parties involved in the Middle East who are not talking to each other now."

Formal political discussions with the Arab League would put the Europeans in virtual diplomatic contact with the Palestine Liberation Organization, which takes over the presidency of the Arab League in September.

Otherwise, the contacts could be made by individual European leaders and diplomats with their counterparts in the Middle East. French President Valery Giscard d'Estaing and British Foreign Secretary Lord Carrington have both discussed the Palestinian question on trips to the Persian Gulf this year.King Hussein of Jordan also has been in close contact with British officials to push for greater European involvement in Middle East.

"We are not seeking to set up our own negotiations," said one British source. "We're saying, 'let's talk to individuals involved and see if there is some common ground to build on.'"

Sources here and in Europe stressed that the European leaders were not planning to formally recognize the PLO, although there may be a reference to the need to include the PLO in Middle East peace efforts in the European Community's Venice declaration.