The United States and Britain ended three days of talks here with Rhodesian nationalist guerrillas yesterday without resolving serious differences over cease-fire terms and over precise arrangements for an interim government during transition to black majority rule.

It seemed that the main result of the conference was to finally set in motion a negotiating process between the two western powers and the black nationalist guerrilla alliance, the Patriotic Front, that could eventually lead to an agreement. But even if this were to happen, the problem of how to get the white minority Rhodesian government to accept it would remain.

Leaders of the two sides gave differing statements at the end of the conference.

British Foreign Secretary David Owen spoke of "increased understanding" and of "progress but not agreement" in the first substantive talks the two western powers have held on their Rhodesia peace plan with the externally based Patriotic Front since the plan was published Sept. 1.

Both U.S Ambassador Andrew Young and the Front's leaders, Robert Mugabe and Joshua Nkomo, had a far more upbear attitude toward the results of the conference.

"I think in terms of creating an overall framework in which we can work together (with the Front) in a climate of trust, we have made remarable progress," Young told reporters at a joint British- American press conference.

The two Front leaders issued a statement later describing the talks as "substantive, frank, comprehensive and conducted in an atmosphere of seriousness." The two parties, the statement said, achieved a "higher level of understanding" of each other's position and alo agreed that arrangements for the interin period were "central and crucial" to reaching an accord.

The two sides have agreed to continue their talks but the place and time were not immediately set.

Owen made it clear that he still wants to bring the three other nationalist leaders presently engaged in separate talks in Salisbury with Rhodesian Prime Minister Ian Smith into a single overall accord acceptable to all parties.

Hinting strongly at direct contacts soon with the internal nationalist leaders, Owen said, "There is no question on our part of not being prepared within the framework of the Anglo-U.S. proposals to talk to anyone."

The three nationalist involved in the Salisbury talks are Bishop Abel Muzorewa, the Rev. Ndabaningi Sithole and Chief Jeremiah Chirau.

It was apparent from Owen's comments that he was disappointed with the results of the conference and upset that the Front had not budged from what he called "certain obviously unshakable principles" regarding its own political role and that of its guerrilla forces in the transition period.

It seemed that the British foreign secretary felt himself under great pressure from critics aof his Rhodesia diplomacy at home and had hoped to bring back more concrete results to defend himself.

Although neither side made its negotiating position public, the key sticking points and the areas for possible compromise became clear by the end of the conference.

Under the Anglo-American peace plan, a British-appointed resident commissioner designate, Field Marshal Lord Carver, would rule the country alone with sweeping powers after Smith "surrenders" and hands back power to Britain, the former colonial authority.

With the backing of a U.N. peace keeping-force, the British would then organise "Free and impartial elections" based on adult universal suffrage in which all black nationalist factions would participate along with the remaining whites to choose a black majority government.

The Front counterplan calls for the creation of a "governing council," which its leaders described yesterday as a "kind of partnership" between themselves and the British with no apparent role for the three internal nationalist factions.

Although the Front accepts a "supervisory role" for Britain in this council, Mugabe said it also wanted a substantial role for itself.

Mugabe also made it clear that while the Front accepts the Anglo-American proposal for elections in the interim period, it rejects the idea that Britain alone should be responsible for supervising them. He suggested that the United Nations or the Organization of African Unity, or it could be Malta for that matter," should supervise the elections.

As for the highly emotional issue of the role of the guerrilla forces in the interim period, the Front leaders insisted that they were "the sole guarantor of the irreversibility of the transitional process" toward black majority rule.

The only Front demand to which the British and American negotiators acceded was the formation of a "governing council." No agreement was apparently reached on the council's powers or composition, however.

U.S. sources also said that the council, as envesaged by London and Washington, would have no authority over military issues and that the key portfolios would be held by the British resident commissioner rather than any of the nationalist representatives.