A still-secret British-American plan for transfering power to Rhodesia's black majority rules out any leadership role for Prime Minister Ian Smith, according to informed sources.

It calls for major changes in the military balance, dismantling Rhodesian government and black nationalist guerrilla forces and bringing an international peacekeeping force into the country.

The outlines of the plan, which is likely to run into difficulties both from the white Rhodesian government and from the guerrillas, were learned here today following a 2 1/2-hour meeting yesterday between Andrew Young, U.S. ambassador to the United Nations, and Lt. Gen. Olesegun Obasanjo, head of Nigeria's ruling military council. The Nigerians would play a key role in the plan by providing the bulk of the troops for the international force.

Young and British Foreign Secretary David Owen plan a series of meetings next week to try to gain approval of the plan from all parties involved in the conflict. It seems doubtful that Smith will agree to step aside voluntarily.

[In Salisbury, a Rhodesian Foreign official called the plan "cynical" and "totally unacceptable."]

Apparently the British and Americans are counting on the presidents of the five "front-line" states, the black African countries most involved in trying to end white rule, to bring pressure to bear on the major black nationalist groups to accept the settlement.

The Patriotic Front, the main guerilla group doing the fighting, is expected to oppose the plan because it would eventually be disarmed under the proposal. It has sought sole resposibility for military security during a transition period.

If the front-line leaders accept the plan, however, most observers feel that there is very little the Patriotic Front can do about it, because the guerrillas depend on the five countries - Tanzania, Zambia, Botswana, Mozambique and Angola - for bases, training and equipment.

At the plan's heart is the establishment of an international military force, most likely along the lines of past United Nations peacekeeping forces. It is said to involve thousands of troops, with most of the forces coming from oil-rich Nigeria, Africa's most populous state and a major contributor to the coffers of the nationalist guerrillas fighting in Rhodesia.

During a transition leading up to the handover of power by th end of 1978, the international force would work alongside elements of the Smith government's white-led armed forces and of the rival black nationalist guerrillas groups.

It was learned that black Rhodesian troops trained and equipped in Tanzania are also to play a major role in the transition, according to the plan.

Said to number 5,000, they constitute an indenpendent force in Tanzania outside the control of the Patriotic Front's guerrillas of Joshua Nkomo and Robert Mugable - themselves said to be at odds.

The Americans and British hope that the outside international force - and to a lesser degree the troops supported by Tanzania's President Julius Nyerere - will help to insure the political neutrality necessary for all Rhodesians to complete for power peacefully and fairly.

As such, the plan is designed to guarantee law and order for the white minority and a fair shake for two other black nationalist factions, headed by Bishop Abel Muzorewa and the Rev. Ndabaningi Sithole. They have no troops but are thought to command considerable support among Rhodesian blacks.

The international force would also try to build a new army for the country. At stake is dismantling not only of the black nationalist forces, but also crack units of the Smith government's armed forces, such as the mixed black and white Selous Scouts, a widely feared counter-terrorist commando unit.

It was learned that there was a "high" likelihood that Britain would serve as overall administrator to prepare elections for what the plan calls a new "non-racial" government.

Carefully balancing its advantages for Rhodesia's whites and blacks, the plan satisfies the nationalists' demands for one-man, one-vote. But it also calls for an enlarged outside investment fund designed to build white confidence in a future Rhodesion economy.

Instead of promising $1 billion to buy out the 270,000 white settlers, as former Secretary of State Henry Kissinger outlined last year, the plan calls for funds to encourage whites to stay and to strengthen the Rhodesian economy. The longer the whites stay, the more they stand to get paid under the plan.

Leading non-Communist industrialized nations - as well as members of the Organization of Petroleum Exporting Countries - have been asked to contribute to the fund.

Running throughout the Anglo-American initiative, it was learned, was a determination that "no one has a lever."

Owen and Young are scheduled to arrive in Lusake, Zambia, this week to brief the front-line leaders on the plan. They are expected to travel to Rhodesia and South Africa after their meeting in Lusaka, the Zambian capital.

South African Foreign Minister R. F. (Pik) Botha, who received the broad outline of the plan several weeks ago in London, has conferred twice since then with Smith. A meeting is planned in Pretroia Saturday between Smith and South African Prime Minister John Vorster.