Western European leaders today jointly declared that the Palestine Liberation Organization "will have to be associated" with the Middle East peace process.

They also announced that they would begin soon to make diplomatic "contacts" with the PLO and other Arab parties to prepare for a possible major European initiative.

Together, these statements represent the nine-nation European Common Market's boldest assertion of a special role in the Middle East, reflecting its mounting concern that President Carter's Camp David peace process may have irretrievably foundered.

The Europeans declared that the Palestinian problem "is not one simply of refugees" but also a matter of the self-determination of the Palestinian people. They stopped short of making a reference to a Palestinian state, however.

The purpose of the European talks with Arab parties to the Palestinian issue will be exploratory, although it was unclear what the European leaders expect to learn about Arab views on a Middle East solution that is not already known. The deeper aim of the mission appears to be to set Europe up for its own major peace initiative at some later date.

While the Europeans bowed to U.S. pressure not to push for changes in key U.N. resolutions, the united European stand on the Middle East nevertheless represents yet another point of friction between Washington and its closest allies.

Following the precedents of the European responses to the Soviet invasion of Afghanistan and the Iranian hostage issue, it demonstrates a continuing lack of ability on the part of the Carter administration to win its way on major foreign policy issues.

In a separate statement on the Soviet invasion of Afghanistan, the European leaders stressed the "genuinely national nature of the resistance" offered by the Afghan people and renewed their call for a "neutral and nonaligned" Afghanistan.

Without U.S. endorsement and Israeli acquiescence, any European diplomatic initiative in the Middle East is likely to have little direct effect. But today's action could pose problems for Carter by irritating the Israelis and complicating already highly sensitive diplomatic maneuvering in the peace process.

Defending their three-page declaration, European leaders said they did not intend to interfere with Carter's Camp David effort to revive stalled Egyptian-Israeli talks on autonomy for Palestinians in Israeli-occupied territory. They described their new diplomatic venture as paralleling the president's own.

"We don't want to compete with Carter," Francesco Cossiga, the Italian prime minister and current president of the European Council, told reporters at the close of a two-day European summit.

Later, in answer to questions, Cossiga added, "This initiative is not meant to be on a collision course with, nor in support of, the Camp David process. Rather, it would be alongside, involving contacts with [Arab] parties outside the current process."

British Prime Minister Margaret Thatcher told reporters, "What we're doing here is supplementing U.S. efforts."

How the Europeans plan to proceed from here, however, was left vague, reflecting differences of opinion among the European states. Elaborating on the official statement, with the help probably of other high-level European officials and probably beginning at the end of this month when the one-year-long council presidency switches to Luxembourg.

Under pressure from Washington, the Europeans dropped early draft proposals, pushed by the French and British, that called for a new U.N. Security Council resolution to gain international recognition for the right of the Palestinians to self-determination and of the PLO to participate in the peace process. Carter said he would veto such a resolution. Since 1967, America's peace-making efforts have been based on Resolution 242, which speaks only of Palestinian refugees.

While calling for involvement of the PLO in the peace process, the language of today's statement was left vague. The council rejected a proposed French-Italian phrasing that would have called for "participation" of the PLO in the peace talks.

Even so, such a statement is likely to disturb the Israeli government, which strongly opposes diplomatic recognition or contacts with the PLO, regarded by Tel Aviv as a "terrorist organization." The United States has generally adhered to this shunning of the PLO, but the Europeans do not believe the PLO can realistically be excluded.

Additionally, the European chiefs reaffirmed the right of Israel to exist and to its security, and declared they are prepared to help guarantee that security. At the same time, they condemned Israel's settlements policy as a "serious obstacle to the peace process."

Also, picking up on a West German proposal, the leaders of the nine Common Market countries called on all parties to renounce the use of force.

While foreign ministers were preoccupied yesterday and through tonight on the exact test of the Mideast statement, European heads of governments spend most of their time talking on other matters, primarily the world economic outlook and East-West and North-South relations.

In their final communique, the European leaders reaffirmed their major short-term economic objective of containing inflation. And looking toward next week's world economic summit here, they reported their concern about the worsening international energy crisis.