Rhodesia's multiracial leaders yesterday rejected as "doomed to certain failure" the British-American proposal for a conference including militant guerrilla factions opposed to the new government here.

In an interview with The Washington Post, Prime Minister Ian Smith said he and his three black colleagues in the ruling Executive Council had sent a message to Britain and the United States telling them they were "on the wrong track" in seeking to hold such a conference.

"We believe they would be well advised to direct their energies into considering the very real and constructive success we've achieved in our agreement here," he said referring to the accord signed in Salisbury March 3 between his government and leader of three black internal groups.

While a statement issued here last night by the ruling Executive Council conferences involving the guerrilla leaders, it said that a meeting "on the lines suggested would have no more prospect of success" then the last all-party Rhodesia conference held in Geneva in late 1976.

"We therefore urge the two governments to reexamine their policies in the light of the radically changed circumstances in Rhodesia,' the statement declared.

The decision taken by the month-old transitional government appeared deliberately designed to force Britain and the United States into abandoning their own proposals for a Rhodesian settlement and pursuing instead new negotiations based on the Salisbury agreement.

It brings to a virtual impasse the Anglo-American initiative over Rhodesia, personally pressed here last week by Secretary of State Cyrus Vance and British Foreign Secretary David Owen, and aimed at averting a civil war among contending black factions. It is widely feared that such a war would serve to embroil the Soviets and Cubans in yet another African conflict, this time on the side of the guerrilla alliance, the Patriotic Front.

Despite their defiant stand, the Rhodesian black and white leaders left the door open for the Front leaders to return to Salisbury."It remains open provided they undertake to operate peacefully," they said in a statement released here last night.

None of them, however, showed any willingness to make special concessions to entice the guerrilla leaders into returning home.

Both Prime Minister Smith and two the black leaders on the Executive Council, Biship Abel Muzorewa and Rev. Ndabaningi Sithole, expressed considerable confidence that the transitional government will be able to bring about a cease-fire in the guerrilla war and carry out elections for a black majority government.

Smith, who tenaciously held onto white rule for 12 years in defiance of the world, is now just as adamant in defending the new multiracial government. Arguing that the Salisbury agreement had satisfied all the basic British and American principles during years of negotiations to get him to accept black majority rule, Smith declared, "We wonder why, just because its a little bit later now, the conditions for Anglo-American support had changed."

This same spirit was reflected in the Executive Council statement which said:

"The members of the Executive Council are united in their determination to proceed as rapidly as possible. . . . They are also united in their resolve not to reopen negotiations on matters which have already been decided."

Smith called upon the two Western powers to "use their good offices" to bring some economic relief to this hard-pressed nation through the lifting of sanctions imposed by the world community a year after the whites' unilateral declaration of independence from Britain in 1965.

He said this would allow the transitional government to hand over the country after the forthcoming elections in good economic health.

On the critical issue of the continuing guerrilla war, Smith, Muzorewa and Sithole each said in separate interviews that a carefully prepared call for a cease-five would be issued in a matter of weeks. They indicated that they expected many of the 5,000 to 7,000 guerrillas now inside the country to lay down their arms and support the internal settlement.

Sithole in particular said that "by the end or the month it will be obvious that the military situation is under control. We are in physical contact with the guerrillas . . . My Lieutenants are meeting with them in the bush. Increasingly, the guerrillas are expressing their solidarity with us."

Regarding the return of the guerrilla leaders, Sithole said that he thought it likely that Joshua Nkoma, coleader of the Patriotic Front, would eventually find some formula under which he could make his way back into country. But he expressed no such optimism about the Front's other, more radical, coleader, Robert Mugabe.

The United States had been courting the Front, particularly Mugabe, in the hope of getting the guerrilla leaders to the negotiating table with the internal black leaders and arranging elections in which all would participate to decide the leadership issue. This now seems an increasingly remote possibility.

The decision to reject the British-American proposal was taken yesterday morning at a meeting of the Executive Council Smith later said there had been no dissension over the issue and comments by Muzorewa and Sithole seemed to confirm that there had been broad agreement in perhaps the most fateful decision yet taken by the new government.

Reflecting on the changes that have taken place with the establishment of the new multiracial government, Smith said he believed that the 12 years of his holdout against the world had prepared the country well for the present situation. A decade ago, he said, "we (the whites) would have given ground and gotten nothing back in return . . . because we went on for so long we got blacks to a stage where they have come to a conclusion that the best things they could do . . . was deal with us."

For the first time in the history of Rhodesia, he said, a majority of black people have come to an agreement with a majority of whites and without the long period of black-white negotiations I don't believe we would have ever solved our problem."