Rhodesian Prime MInister Ian Smith and three moderate black natiolist leaders signed a historic agreement yesterday bringing to an end nearly 90 years of all-white rule in this breakway former British colony and paving the way for the establishment of a fully black-run government at the end of this year.

While all parties to the agreement praised it, several major hurdles lie in its path. Among them are approval by the country's white-dominated Parliament and its African population.

In addition, there is no assurance that the accord will bring an end to the steadily worsening guerilla war since the Patriotic Front that is doing the fighting has denounced the entire scheme as "the biggest sellout in African history" and vowed to fight on until total military victory.

The United States and Britain reacted cautiously to news of the settlement, and in a clear reference to the Patriotic Front. U.N. Secretary General Kurt Waldheim warned that the Rhodesian problem would not be solved "as long as some are excluded from that process."

The solemn signing ceremony was held in a packed side room of the red brick mansion where the negotiations took place and that formerly served as the residence of the governor of Southern Rhodesia. That was the name of the country before the whites declared their unilateral independence from Britain in November 1965.

Later one of the black leaders, Chief Jeremiah Chirau, told a crowd of reporters gathered on the mansion's front lawn that "today we have got what the people of this country have wanted for many years."

The complicated, six-page accord reached after three months of hard negotiations provides for Smith to remain as prime minister in a new multiracial interim government and as a member of the four-man executive council, including the three black leaders, that will lead it. Decissions are to be made by consensus, assuring both sides of a veto power.

In addition to the executive council, the accord calls for a ministerial council that will have an equal number of black and white ministers.

The agreement effectively spells the end of the rule in Rhodesia but insures that the 268,000 whites will continue to play a dominant role in both the interim government and the writing of a new constitution for Zimbabwe, the name this country will undoubtedly take once the 6.7 million blacks take power on Dec. 31.

Smith said the exact date for the transitional government to begin functioning would be decided shortly by the four leaders jointly, as well as the question of whether a nationwide referendum of both the black and white populations would be held to approve the agreement.

The accord came after three other abortive attempts by Smith in the past four years to negotiate an internal settlement directly with Rhodesia's black leaders. Two of these attempts involved the two other blacks involved in the present negotiations, Bishop Abel Muzorewa, leader of the United African National Council and Rev. Ndabaningi Sithole, head of his own African National Council.

Probably the most difficult task immediately facing the new interim government will be arranging for a ceasefire in the war and convincing the 15,000 to 20,000 guerrillas to defy their patriotic front leader and accept Salisbury's authority.

While the three black leaders predicted yesterday that "the overwhelming majority" of the guerrillas would back the settlement, there remained considerable doubt among these outside observers.

Smith again reiterated his position at a press conference following the signing ceremony that "anybody who wishes to come back and work peacefully to produce a new constitution for Rhodesia is entitled to do so, provided they forego terrorism."

He said, however, that the interim government would go on fighting if the Patriotic Fron did. "Remember they will be fighting against a majority government in this country," he remarked. "I just don't believe that they will receive the support of Rhodesians."

The other main functions of the transitional government, according to the agreement, include consideration of the release of political detainees, the removal of racial discrimination laws, the organization of "free and democratic elections" and the drafting of a new condsitution.

The agreement also embodies "certain fundamental principles" for the constitution that were agreed upon by the four leaders 16 days ago. These include majority rule based on universal adult sufferage, a 100-member Parliament with 28 seats reserved for the whites, and various safeguards for the white minority such as an independent judiciary and a bill of rights.

Bishop Muzorewa, who left almost immediately after the signing for a meeting in London with British officials, made an impassioned plea for Western recognition of the interim government.

"They must recognize us. This is no longer UDI," he said referring to the whites unilateral declaration of independence in 1965 which neither Britain nor any other nation including South Africa has accepted as legal.

He said he was still uncertain whether he would participate in the U.N. Security Council debate on the whole Rhodesian issue, scheduled for next week. He said that if he was asked to address the council, "I will speak like I never have spoken before."

The white minority government here is counting heavily on the three black leaders to sell the agreement both to the Western powers, particularly Britain and the United States, and to international bodies like the United Nations and the Organization of African Unity.

Both Muzorewa and Sithole are planning trips to Washington for this purpose and there is a strong possibility Muzorewa will travel on from London to Washington next week.

At the joint news conference, Smith described the interim government as "a king of halfway stage" to the final step which would come only when the new constitution is passed by the present white-controlled Parliament.

He said all four leaders would henceforth be making decisions collectively, including whether to hold a national referendum on the whole agreement.

On the whole, the black leaders, particularly Muzorewa, seem to have given in to Smith on many of their key demands. For example, the bishop had been insisting on a neutral, outside chairman at the head of the executive council and a three-quarters majority for the blacks in the ministerial council.

Instead, the executive body is to have a rotating chairman chosen from among the four members and the ministerial council is to consist of an equal number of blacks and whites. In fact, each ministry is to have co-ministers, one black and one white, as Smith had been proposing.

Still, the reaching of any agreement between Smith and the country's black nationalist leaders represents a major achievement.

Smith began on a search for such a settlement as far back as the summer of 1974, holding secret talks first with Muzorewa. He later claimed they actially reached an agreement that the bishop signed but later reneged on after consulting with his organization's executive.

Again in the summer of 1975, Smith met with Muzorewa. Sithole and Joshua Nkomo, coleader of the Patriotic Front, in a train stationed on the bridge spanning Victoria Falls between Rhodesia and Zambia. That meeting, arranged by South African Prime Minister John Voster in cooperation with Zambian President Kenneth Kaunda, aborted before any substantive talks really got under way.

Following this failure, Smith convinced Nkomo to hold talks alone with him in late 1975, and these two leaders continued their negotiations until March 19, 1976 when they collapsed amid much bitterness on both sides.

The agreement Smith has just accepted, however, goes much farther in making major concessions to the blacks than any of the previous negotiations, none of which would have brought black majority rule to Rhodesia in less than several decades, if then.

In addition to Smith's internal talks with various black leaders, both the British and Americans made attempts at promoting a settlement. All failed.

One of the attempts was engineered by former secretary of state Henry Kissinger, who was the first person to force Smith to accept the principle of majority rule back in September 1976. The negotiations that followed under British auspices broke down.