Prime Minister Ian Smith and three African nationalist leaders announced yesterday that they had reached agreement on a formula for black majority rule in Rhodesia.

The agreement on the major points of a new constitution climaxed 10 weeks of talks in which Smith sought to work out an "internal settlement" with three black Rhodesians leaders who agreed to negotiate with him.

Sources said the Smith government and the black leaders would probably complete within a week or two details of a transitional administration that will finally bring an end to white minority rule in this breakway British colony.

While Smith hailed yesterday's agreement as "a victory for moderation," Rhodesian guerrillas based outside the country rejected the proposal out of hand.

"It will not work," declared Joshua Nkomo, who along with Robert Mugabe, leads a Patriotic Front alliance of 40,000 guerrillas based in neighboring Tanzania and Zambia.

"The war continues," Nkomo told newsmen in Lusaka, Zambia. "We now know who the enemies are."

The United States, which along with Britain has been attempting to work out a settlement between Smith and the Patriotic Front, also expressed strong reservations over the internal accord.

U.N. Ambassador Andrew Young said in New York that a settlement which excluded the Patriotic Front could lead to "another Angola-type situation" in which blacks wind up fighting blacks in a bloody civil war.

That prospect would also raise the possibility of increased Soviet and Cuban involvement.

"This is not a settlement addressing itself to the issues for which some 40,000 guerrillas are fighting," Young declared.

Nevertheless, Smith and the three blacks who joined him in the talks were jubilant yesterday as they strode side-by-side across the lawn of a suburban mansion to make announcement.

"We are very happy," said Bishop Abel Muzorewa, whose United African National Council had deadlocked the talks for three weeks over the ques tion of white-minority representation in the Parliament of the new nation that will be renamed Zimbabwe.

When the leaders were asked by newsmen how soon a transitional government could be formed, Chief Jeremiah Chirau, leader of the Zimbabwe United Peoples Organization, replied: "It could be days."

The third organization that participated in the internal settlement talks, the African National Council was represented yesterday by Elliot Gabellah, a stand-in for the Rev. Ndabaningi Sithole.

Sithole departed for Europe earlier in the day to seek support for the settlement in Belgium, France, Britain and West Germany.

The final agreement announced last night calls for a 100-member national assembly elected by all adults above the age of 18. For 10-year period, however, whites - a minority of 263,000 in this country of 6.8 million blacks - would be guaranteed 28 seats. This would enable whites to block any effort to annual constitutional guarantees that will protect them against arbitrary seizure of their land or property.

One of Smith's major objections to the Anglo-British proposals was that they lacked specific provision for a white bloc in the new Zimbabwe Parliament.

The formula announced yesterday for how the 28 white representatives will be elected did represent a compromise. While a purely white voting roll will elect 20 of the members, the other eight white representatives will be elected by both black and white voters but from a list of candidates nominated by the conservative white Rhodesian front party.

The other key point that Smith objected to in the Anglo-British plan was that it called for the British - whom Smith ousted in 1965 when he unilaterally declared the colony independent - to preside over the transition.

The interim government envisaged under the internal plan, sources said, will be two-tiered led by a council of state composed of Smith, Sithole Bishop Muzorewa and Chief Chirau.

They will be assisted in running the country during the transition by a Cabinet that will include black ministers, the source said.

A number of observers were puzzled by Bishop Muzorewa's seeming about face in dropping his vociferous demands and agreeing to the formula announced yesterday.

Only Sunday, the bishop had indicated a hardening of his stand on an interim government which suggested protracted negotiations still lay ahead even after the white representation issue was solved.

Until today the bishop was thought holding out for prior agreement on key issues - such as arranging a ceasefire, release of political and other detainees, integration of nationalist guerrillas into Rhodesia's security forces, dismantling of anti-black legislation and drafting of a futire constitution. Also implicit was Smith's departure as prime minister.

Political analysts offered the following explanations for the bishop's change of heart:

His inner circle is willing to pay any price to get Smith out of office.

They feel that they can change the rules once they are in the driver's seat.

A quick agreement would prevent their arch-rival, Joshua Nkomo, from returning from Zambia and contesting the elections.

Smith is said to still hope Nkomo would return to become the country's first black prime minister with white and other small party support.

Although considered the father of African nationalism in this country, Nkomo generally is not thought able to muster enough support to emerge as leader of the biggest parliamentary bloc in free elections.

Pessimistic observers feared, however, that the bishop's decision will only increase the guerrillas' ardor and accelerate the war without gaining the vital British and American support needed to lift the sanctions which increasingly hurt Rhodesia's enfeebled economy.

The next test of the bishop's attitude is expected when the negotiators tackle the nature of the transition government. Emboldened by his coup on white representation, Smith was thought likely to try to drive a hard bargain.

The other major elements of the guarantees for the while minority, agreed upon soon after the talks began in early December, call for:

A bill of rights backed by the legal system.

Entrenched clauses in the constitution for the first 10 years of black majority government which can only be overturned by a vote of 78 of the 100 members of Parliament.

Guaranteed pensions for white civil servants.

Guarantees of white civil service jobs - including the armed services and police.

Universal suffrage at age 18.