The Organization of African Unity's summit ended peacefully early today after a compromise was reached on the controversial Egyptian-Israeli treaty. An explosive Tanzanian-Ugandan debate remained unresolved, however.

The African leaders who attended also agreed on a potential formula for ending the three-year war over the former Spanish colony in the Western Sahara. They agreed in principle to establish a pan-African defense force and to work up an African human-rights charter.

Ending one of the mos tacrimonious OAU summits in recent history, Liberian William Tolbert told about 20 African leaders that divisive African issues must be faced with candor. In the past, the generally conservative organization has sought to avoid confrontations between member states and defuse disagreements with broad resolutions attained by consensus.

During Idi Amin's rule in Uganda, most African leaders remained silent on reports of hundreds of thousands of deaths there.

But Tolbert indicated in his speech that this method of handling the continent's affairs should stop.

"We cannot permit our quest for freedom to be impeded by fraternal hostilities," Tolbert said in his 3 a.m. speech closing the conference.

"It is most important in our times to answer the issues of fractured friendships and distractive disunity with answers of genuine accommodation and reconciliation," said Tolbert, who will be chairman of the OAU for the coming year.

The four-day conference, attended by delegations from 48 African countries, was punctuated by harsh recriminations and angry walkouts over an Arab-led effort to condemn the Egyptian-Israeli peace treaty and over Nigerian and Sudanese criticism of Tanzania's toppling of Ugandan dictator Idi Amin. Morocco walked out over a self-determination resolution on the Western Sahara, which Morocco claims.

North African Arab delegates, led by Libya and Algeria, had come to the summit seeking the expulsion of Egypt from the OAU because of its treaty with Israel. Egyptian President Anwar Sadat had made an impassioned defense of the treaty here on Wednesday, and some Arab delegates walked out. His speech was applauded by most of the black African delegates.

The Egyptians considered their position particularly important because Arab League members have taken steps to expel Egypt and the Islamic conference in Morocco did so last May. Boutros Ghali, Egyptian foreign minister, said in an interview that Egypt does not want to be pushed into isolation from such organizations.

After two weeks of secret and bitter debate, the OAU delegates passed a mild resolution upholding "the right of the Palestinians to a homeland," but did not condemn the treaty, said OAU Secretary-General Edem Kodjo.

Morocco's delegation walked out of a closed session Thursday night after delegates voted to support a referendum in the Western Sahara.

Since Spain ceded its Sahara colony to Morocco and Mauritania in 1976, Algerian-backed guerrillas from the territory have been fighting a desert war against the two countries.It was unclear today whether Morocco, which has made historical claims to the territory, will allow a referendum. Mauritania supported the referendum resolution.

The referendum would allow the Western Sahara a choice between independence or continued division of the territory between Morocco and Mauritania.

The summit leaders also voted to:

Establish a pan-African defense force for use in cases of attacks from outside Africa and in intra-African disputes.

Draft an African human-rights charter and create an office within the OAU to monitor the human-rights perfomance of member states.

Hold an extraordinary economic summit in Lagos, Nigeria, in several months, on the establishment of an african economic community.

Recognize the guerrillas of the Patriotic Front as "the sole representatives" of the people of Zimbabwe-Rhodesia but avoid closing the door on a peaceful solution to the war there through an all-party conference, including the government of Bishop Abel Muzorewa. CAPTION: Picture, Liberian President William R. Tolbert, AP