PARIS, OCT. 5 -- Amadou Mahtar Mbow, UNESCO's controversial director general who said last October he would not seek reelection, has emerged as a leading candidate for another six-year term.

Mbow's election to an unprecedented third term as head of the United Nations Educational, Scientific and Cultural Organization would make it highly unlikely that the United States or Britain would return to the agency and could lead to withdrawal by such other countries as Japan, the Netherlands and Canada, a European diplomat said.

The Reagan administration pulled out of UNESCO in 1984, charging that it has been grossly mismanaged and antiwestern under the stewardship of Mbow, a Senegalese.

Britain and Singapore followed suit the next year, leaving UNESCO with a total loss of 30 percent of its $150 million budget. The departure of other countries could chop another third from UNESCO's budget, raising serious questions about the agency's ability to operate effectively or even to survive.

Mbow, 66, told delegates last fall he would refrain from soliciting another term in an effort to free the agency from such political controversy, according an account of a closed meeting provided then by his spokesman, Doudou Diene. This was welcomed by U.S. and other officials as a statesmanlike gesture that would allow UNESCO to make the reforms demanded by Washington and London as conditions for their return.

But last month, Mbow's home government in Dakar and the Organization of African Unity nominated him for reelection. Reports here said Mbow, the first African to head a major international organization, lobbied African and Arab governments during the past year to secure their support for staying in his $170,000-a-year job.

UNESCO's 50-member Executive Board, meeting this month at the agency's Paris headquarters, is scheduled to vote on the next director general beginning Tuesday. If no candidate has won after four secret ballots, the board chooses between the top two in a final round of voting.

Mbow is considered the best placed of about 10 candidates, a diplomatic observer said. This is largely because African countries form the biggest bloc on the Executive Board, followed by Arab countries, whose governments also have supported Mbow in the past.

France and several other major countries have announced their support for another candidate, Foreign Minister Sahabzada Yaqub Khan of Pakistan. The French decision was considered particularly important because UNESCO is headquartered here and France traditionally has sought to promote the interests of French-speaking African countries such as Mbow's Senegal.

"Yaqub Khan is the one who has the best chance of creating unity to enable UNESCO to get by this difficult moment," a French official said.

It was unclear whether France also plans to use its considerable influence with African countries to steer their votes away from Mbow. Prime Minister Jacques Chirac went out of his way to salute Mbow last November for declaring he would not seek reelection, and the French official made it clear Mbow's departure is seen here as a good idea.

Some countries, particularly in Latin America, have expressed reluctance to support Khan because he was a career military officer before entering diplomacy and as foreign minister has served President Zia ul-Haq, a fellow officer who took over Pakistan in a military coup. In addition, some reports have tagged him as the "American candidate," underlining the North-South divisions left by Mbow's management style and the Reagan administration's pullout.

The Executive Board's decision must be endorsed by the full General Conference, which begins a one-month session here on Oct. 20. This approval traditionally has been routine since UNESCO's founding in 1946 to promote educational, cultural and scientific exchanges. But diplomats suggested this time it could be crucial if the Executive Board selects Mbow and France or other countries seek to reverse the decision.

Under Mbow, UNESCO has been accused of failure to carry out its mandate and of an antiwestern slant. U.S. officials said they saw the latter in his decisions. The critics say that the organization has been spending 80 percent of its budget in the Paris headquarters rather than in the poor countries it is supposed to target.