President Carter and Egyptian President Anwar Sadat yesterday launched a search for Mideast compromises that, if successful, will lead to negotiations here next month on a self-government system for Palestinian inhabitants of Israeli-occupied Arab lands.

The aim of Carter's talks with Sadat, and with Israeli Prime Minister Menachem Begin next week, is to pave the way for a final burst of negotiating activity designed to bring the Palestinian autonomy agreement close to completion by its May 26 target date.

Reliable sources said the idea is for the three leaders to agree on new instructions for their top negotiators -- U.S. special ambassador Sol M. Linowitz, Egyptian Prime Minister Mustafa Khalil and Israeli Interior Minister Yosef Burg -- and have them assemble here in late April or early May to begin what they hope will be the final round of autonomy talks.

In earlier phases of the negotiations, the three have met in Israel. Egypt or Europe for two- or three-day periods to review progress of their subordinates and to make quick passes at breaking the many disagreements that have stalled the 10-month-old talks.

Under the tentative new plan, sources said, the three are expected to have considerably wider negotiating latitude and will be under orders to keep the talks going until they either reach agreement or are forced to report failure.

In this respect, the sources noted, the talks are intended to follow the so-called Blair House model used in late 1978 and early 1979 to do the major negotiating on what eventually became last year's Egyptian-Israeli peace treaty.

In the Blair House round, top Israeli and Egyptian cabinet ministers met in Washington almost continuously under the mediation of Secretary of State Cyrus R. Vance.

However, the sources cautioned, a move to negotiations under Linowitz's mediation will depend on whether Carter, through his talks with Sadat and Begin, can find enough flexibility in the Israeli and Egyptian positions to suggest areas for compromise.

Whether that can be done was still an open question yesterday when Carter and Sadat held the first two of their three scheduled meetings. They talked three hours and are scheduled to meet again this morning.

At issue are Egypt's demand that the West Bank and Gaza Strip be given self-governing rights and Israel's determination to limit those powers so severely that the area could not become the nucleus of an independent Palestinian state.

At a state dinner for Sadat last night, Carter seemed to be leaning toward the egyptian argument that the talks call for full autonomy for the Palestinians.

"Prime Minister Begin said many times in the presence of President Sadat and me, not just autonomy -- full autonomy. 'Full autonomy,' he said many, many times," Carter recalled.

Carter has an enormous stake in the automy talks; they involve the success or failure of his greatest foreign policy achievement -- the Camp David accords that led to the Egyptian-Israeli peace treaty. The president hopes to see them expanded into a Mideast peace process to end the three-decade conflict between Israel and the Arab world.

In his dinner toast last night, Carter said "it would be inconceivable to me that we would let this promise" of the Camp David accords "slip from our grasp and end hopes and confidence for the aspirations of two troubled peoples."

Sadat also has a lot riding on the outcome. The peace treaty estranged Egypt from its traditional Arab allies and Sadat wants a workable autonomy agreement to counter charges that he made a separate deal with Israel at the expense of the Palestinians.

However, the negotiations have been snagged from the outset by issues that evoke emotion and controversy among both Arabs and Israelis.

In addition, there are major, unresolved disputes about control over land and scarce water resources in the occupied territories, the degree to which Palestinian residents of East Jerusalem would be covered by the agreement and Arab hostility to Israel's insistence on the right to establish Jewish settlements on the West Bank.

In this week's talks, Carter will be hearing Sadat's views on how much he is willing to give in regard to these matters and how much Egypt expects in compromises from the Israeli side. However, U.S. officials cautioned yesterday, the extent of the gap that has to be overcome won't become clear until the president hears from Begin next week.

Even if the White House talks do succeed in setting up a new negotiating round in Washington, many of the parties, including Sadat, are known to believe that a new three-way summit eventually will be required. It would dispose of issues beyond the scope of the negotiators.

However, U.S. officials have been unwilling to concede the inevitability of a new summit. Instead, they said yesterday, Carter wants to pursue a step-by-step approach that would put the negotiators in position to have an extended new round and then see what they are able to do before addressing the question of a summit.

Uri Avneri, of the left-wing Sheli Party, said after returning from Cairo tonight that Sadat talked of giving a "new push" to the peace process by talking directly to the Israeli people.

Avneri said Sadat mentioned no time for a repeat of his historic November 1977 trip to Jerusalem, but that he left the impression it would be after the May 28 target date for concluding negotiations on the proposed autonomy scheme for West Bank and Gaza Strip Palestinians.

Aveneri's account was given some additional credence by independent reports from Cairo and Washington that Sadat would accept an invitation from Begin to travel to Jerusalem again.

Anis Mansour, editor of October magazine and a confidant of Sadat's, told foreign journalists in Cairo to expect a major development arising from Avneri's interview, which is to appear in Haalom Hazeh, a Hebrew magazine, Wednesday. Mansour said it involved a major new initiative.

Also, Shalom Kital, reporting from Washington for Radio Israel, quoted members of the Egyptian delegation there for today's Sadat-Carter talks, as saying the Egyptian president hopes to return to Jerusalem.

Avneri said, "He was extremely serious about it. He talked at great length about it, and it was obvious he had given it a great amount of thought." l

Avneri said in a telephone interview that he began his meeting with Sadat half-jokingly by suggesting the time might be right for another trip to Jerusalem.

"He looked at me in a very curious way, as if I had known something I didn't really know. He said, 'You know, I've been thinking for two weeks exactly about this,'" Avneri said.Sadat had just completed a two-week period of reflection at a country villa.

Sadat reportedly recalled his first visit and how it was designed to convince the Israeli people to relinquish "sacred Egyptian soil" in the Sinai Penninsula.The purpose of the next trip would be to tell Israelis how Arabs felt about Palestinian independence, Avneri quoted the Egyptian leader as saying.