Britain and the United States begin the first substantive talks here today with leaders of the Patriotic Front, the Rhodesian black nationalist guerrilla organization, on the long-stalled Anglo-American plan for a settlement of the worsening Rhodesia conflict.

The three-day conference here has taken on special urgency since reports last week from the Rhodesian capital of Salisbury that Prime Minister Ian Smith and three moderate black leaders are nearing agreement on a multiracial interim government that would bypass the British-American plan.

Should the conference here fail and Smith succeed in forming such a government, the Anglo-American plan would probably be doomed to failure, opening up the prospect of a bloody Angola-style civil war among black nationalist factions that could easily involve South Africa and the big powers.

Thus the Malta talks, first scheduled to take place Nov. 16, have taken on added meaning to all the concerned parties.

Meanwhile, informed British and American sources say both London and Washington last week actively encouraged the three nationalist leaders talking to Smith to hold out for terms more like those contained in the British-American proposals.

These outside pressures are reported to have been responsible for the last minute failure of the Salisbury talks to produce an accord on a multiracial interim government.The announcement had been expected early this week just as the Malta conference was convening, but now this is uncertain.

Both the Patriotic Front leaders, Robert Mugabe and Joshua Nkomo, as well as Britain and America had denounced Smith's attempt to engineer his own internal settlement leaving the front out in the cold.

Their common concern about the growing possibility of such an agreement has given a new impetus to the extremely difficult dialogue between the two Western powers and the guerrilla front over the Anglo-American settlement plan.

In fact, the conference here marks the first time both British Foreign Secretary David Owen and U.S. Ambassador the Unite Nations And drew Young have ment with Mugabe and Nkomo since late August when the two Western diplomats first unveiled the plan to all parties.

Since that time, the British-American proposals for an orderly handover of power from Rhodesia's 263,000 whites to the 6.3 million blacks have met rough going and no real progress has been recorded.

While the five "front-line" African states supporting the front and its guerrilla war have given the plan their backing. Smith formally rejected it Nov. 24 and two weeks later began his own talks with the three black nationalist leaders based inside Rhodesia - Bishop Abel Muzorewa, the Rev. Ndabaningi Sithole and Chief Jeremiah Chirau.

In addition, the front has expressed its strong opposition to a number of key provisions in the Anglo-American scheme, leaving Washington and London in a quandary until recently as to how to get substantive discussion on their own plan under way.

In addition to the front and the two Western negotiators, both the British resident commissioner-Desigate for Rhodesia, Field Marshal Lord Carver, and special U.N. representative Prem Chand are participating in the Malta conference.

The Anglo-American plan calls on Smith to "surrender" power to the British, the former colonial power in Rhodesia, and for Britain then to arrange for "free and impartial" elections on the basis of universal adult suffrage before the end of this year. Free elections could be expected to produce a government representing the black majority to replace Smith's government.

The United Nations would provide a peacekeeping force during the British-supervised transition period lasting no longer than six months. It would also help build a new black-led Rhodesian national army that the British and Americans have demanded be based on the guerrilla forces.

The front has refused to accept the all-powerful role of the proposed British resident commissioner and has demanded that it be permitted to "superintend" the transition period leading up to elections for a black majority government. It is also wary of the presence of a U.N. peacekeeping force in the country and strongly condemns other aspects of the British-American plan regarding guarantees for the white population.

Upon arriving on Malta Sunday night, Nkomo and Mugabe again repeated their demand that the British force Smith to resign and hand over power directly to the "people of Zimbabwe," meaning the Patriotic Front. Zimbabwe is the nationalists' name for Rhodesia. They denounced the three nationalist leaders negotiating with Smith as "puppets" and pledged to continue the war until they topple Smith.

Until late last week, the three black nationalist leaders inside Rhodesia appeared to be on the verge of signing an agreement with Smith. So far, they have extracted from Smith acceptance of universal adult suffrage, but in return they have agreed to reserve 28 of the 100 seats in a new parliament to the whites, and to require a 78-vote majority for any changes of the constitutional guarantees for the white minority.

Other issues such as the composition of the new national army and arrangements for the election of a black government were to be left to the proposed multiracial interim government to resolve.

The issue that reportedly blocked the announcement of a final agreement in Salisbury last week was that of whether the 28 whites would be elected by a separate white voters' roll or a common one composed of both whites and blacks.

The former solution would assure the ruling all-white Rhodesian Front of all the seats, while the latter would make it possible for liberal whites siding with the black majority government to be elected.

In the latter case, it is conceivable that constitutional changes could be made, is, in addition to the 72 black deputies in parliament, six white liberals voted for them.