Africa's quest for political stability and the challenge of South Africa will dominate talks this week at the 17th summit of the Organization of African Unity.

Since last year's meeting at Monrovia, Liveria, the organization to deal with increasingly destructive inter-African conflicts and to halt the economic deterioration that is the root of so much strife across the continent. e

Symbolic of the challenge facing the OAU is the absence at this summit of its former chairman, William R. Tolbert. The Former Liberian President was assassinated when military officers staged a coup in April.

Last year, Tolbert spoke forcefully of the need for African governments to initiate broad policies to protect human rights and wipe away "narrow nationalisms" that hinder regional cooperation.

Tolbert's successor, Master Sgt. Samuel K. Doe, is expected to be accepted reluctantly by the OAU's delegates this week as his country's representative.

The leaders of Sierra Leona, Ivory Coast, Guina and Togo visited Monrovia Friday and met with Doe for three hours. The four are monitoring the new Liberian government's domestic policies with an eye toward ensuring that harsh military rule is not instituted.

As incoming chairman, Sierra Leone's president Siaka Stevens is expected to preside over a somber summit also dominated by such issues as the war between Morocco and Western Saharan guerrillas, the bitter civil war in Chad, the continuing friction between Ethiopia and Somalia and the Arab-Egyptian split on Egypt's troubled peace treaty with Israel.

Stevens has proposed a 15-member body within the OAU to deal with both trans-African and domestic conflicts. The body would resemble the United Nations Security Council.

Sierra Leonean Foreign Minister Abdulai Conteh said the council would be empowered to act immediately, instead of "waiting between scheduled sessions to address [Africa's] crises," as the OAU now must do. s

This week's discussions were held up for two days while Libya and Algeria sought unsuccessfully to deny Egypt its turn to take a vice chairman's seat on the organization's council of foreign ministers.

"Ever since the peace treaty with Israel," Conteh said, "the Arab members [of the OAU] have tried to isolate Egypt." Last year Arab members also failed to have Egypt expelled from the organization.

The Polisario guerrillas of the Western Sahara, who have been engaged in a four-year, hit-and-run battle with Morocco, touched off another furor this week by applying for admission as a member state in exile. Morocco has been battling the guerrillas since Spain ceded its former colony to Morocco and Mauritania in 1976.

Mauritania dropped out of the war last year and Morocco took over its portion of the territory.

Morrocco has threatened to withdraw from the OAU if the Polisario application is approved and has begun intense lobbying to block it consideration. Although more than 20 African states have recognized the Polisario, "their application is unlikely to be completed at this summit" because of Morocco's objections, Conteh said.

A high point of the summit will be the seating of Robert Mugabe, leader of the newly independent Zimbabwe, formerly Rhodesia, which will bring the OAU's membership to 50. Zimbabwe's formal admission will sharpen the organization's focus on South African-ruled Namibia (South-West Africa) and South Africa itself.

The recent South African invasion of southern Angola, ostensibly against the rear base of Namibian nationalist guerrillas, has been an angry topic of discussion here the past week among ministerial delegations. The delegates here voted to ask the Organization of Petroleum Exporting Countries for tighter enforcement of its South African oil embargo.

When Africa's leaders convene on Tuesday there also will be a delicate attempt to bring together two of South Africa's rival nationalist guerrilla movements. The Soviet Union-backed African National Congress, which recently took credit for an effective attack on South Africa's coal-conversion refineries, and the Chinese-backed, Western-favored but divided Pan-Africanist Congress. The two groups have been split since the 1950s.