The Anglo-American effort to find a peaceful solution to Rhodesia's 11-year-old constitutional crisis finally collapsed today with Prime Minister Ian Smith's rejection of the new British settlement plan.

The announcement was made by British negotiator Ivor Richard after his final series of talks with Smith this morning.

The move means the end of an effort begun by former Secretary of State Henry Kissinger last summer to bring Rhodesia's white-minority government and its black nationalists together to discuss terms for the transition to majority rule.

Richard said that the Rhodesian leader had called the plan "so far removed from reality that it would lead to chaos," and unacceptable even as a basis for further negotiations in Geneva with four black nationalist leaders.

Smith announced on television tonight that his government would open new negotiations with unnamed Africans inside the country on the basis of the original Kissinger settlement plan.

As a show of good will and guarantee of intention, Smith said his government would soon remove much of the discriminatory legislation that has allowed whites to hold both economic and political power in Rhodesia.

But the new effort will exclude the four black nationalist leaders, which is certain to make the new settlement effort unacceptable to the outside world.

In his speech, Smith blasted the British plan - a substitute for the Kissinger proposals, which the African nationalists have rejected - as being as different as chalk and cheese from the earlier plan.

The British plan, he said, would "entail immediate black rule, which would be imposed from outside and would in no way represent the views of the majority of black Rhodesians. It would come into being not in two years time, or when the agreed constitutional and electoral processes have been completed, but immediately. It would in no sense be an elected government."

"To put it in a nutshell," Smith said, "the British government wish to impose upon us an interim government, recommended to them by the front-line presidents (of Botswana, Tanzania, Angola, Mozambique and Zambia) working in collusion with the terrorist-oriented Patriotic Front" of militant nationalists Robert Mugabe and Joshua Nkomo.

He added: "We were absolutely clear in our minds that if we accepted these, this would lead almost certainly to a stampede as far as white Rhodesians were concerned . . . In such a country not only would there be no place for the White Rhodesians, moderate black Rhodesians would find themselves an enslaved and persecuted people."

The plan Smith rejected called for a British high commissioner to head the interim government, and a council of ministers with 20 blacks, five liberal whites and five whites from the Smith government. They would have executive powers and would write a new constitution.

In effect, blacks would hold the majority of powers.

Kissinger had proposed a two-tier transition government.

A Council of State with equal representation of blacks and whites, headed by a Rhodesian white, would write a new constitution.

A Council of Ministers would be predominantly black and headed by a black, although whites would hold the two key portfolios, defense and justice. The ministers would have executive responsibility and legislative authority.

The Kissinger plan allowed whites to retain greater powers during the transition period.

Richard laid full blame for the collapse on the Rhodesians, since the black nationalists had accepted the British plan as a basis for resuming the Geneva conference, which adjourned before Christmas after two months of negtiations.

"Mr. Smith alone has rejected our proposals as even a basis for further discussions. He bears, as we see it, a heavy responsibility for what may now follow," Richard said.

He emphasized that the rejected plan has been not put forward as a final settlement, but merely as a new basis for talks: "The details, the fine prints, were all open for discussion, and all open for negotiation," he said.

"At the moment, it is difficult to see where (Smith) can go from here. Fighting will go on. Sanctions will not be lifted. The people of this country, both white and black, will suffer further. The outcome is extremely difficult to foresee," Richard said.

In a private discussion with white Rhodesian farmers Saturday, the British official reportedly painted an even gloomier picture, warning that without a settlement the Cubans would soon join the guerrillas now waging an escalating war on Rhodesia.

He said South Africa would not intervene on Rhodesia's behalf if the guerrillas got outside help, and he predicted that the British air force would be in within a year and a half to airlift whites and "pick up the pieces."

Today Richard would only comment, "I am sad and apprehensive about the future." Richard left for South Africa tonight, his departure delayed a few hours because his laundry had not returned.

In his 45-minute speech tonight, Smith contended: "I have not broken off negotiations. On the contrary, I have urged that we continue to negotiate on the basis of the original agreement to which the British, Americans and Rhodesians were all party. I have assured (Richard) that within the basic framework I am prepared to negotiate with flexibility . . . My government is conmitted to creating a non-racial society in Rhodesia."

The Rhodesian leader said, "I have also urged the British government to change their approach - which has clearly been a failure - and to deal with the Africans in Rhodesia rather than the terrorists and their hosts and sponsors in neighboring countries. Why should a few thousand terrorists, the majority of them mere schoolboys, call the tune to 6 million basically peaceful and peace-loving Africans?"

Smith then announced that he had "issued invitations to the black leaders to join me" in new negotiations based on the Kissinger plan. He did not name the leaders, but added, "Let me make it clear that I exclude from this exercise anyone who supports terrorism" - a clear reference to Nkomo and Uurgake.

He said that black Rhodesians now have a choice: "They either endorse the decision of the front-line presidents, supported by the British government, that we accept the Patriotic Front as the new leaders of Rhodesia. In other words, to invite into Rhodesia a situation comparable to that in Mozambique or Angola. This is the best the outside world can offer.

"The alternative is that Rhodesians come together within their own country and solve the problem among themselves.

Observers here believe that Smith is hoping to win U.S. approval for his move by negotiating on the terms Kissinger outlined in September.

He made an indirect appeal to Washington saying: "The interest and participation of the United States - the most powerful and influential nation on this earth, and the leader of the free world - is something worth retaining."

Smith pledged that his government would move soon to remove legislation that discriminates against Rhodesia's blacks, who outnumber whites by more than 20 to 1.

He did not name the laws to be repealed, but it is widely believed that the first action will be elimination of the controversial Land Tenure Act, which divides Rhodesia into two halves, one for the 270,000 whites, the other for 6 million Africans.

He added, however, that it would be "fruitless" to try to deal with changes in the voting system and Parliament, "because these will be subjects for consideration by the interim government."

He acknowledged that during the interim, the four-year-old guerrilla war will remain a major problem, but said: "The government is at this moment implementing measures to increase our fighting effort. There are additional plans in the pipeline, and speaking in general terms, a new and tougher line is being considered for our security effort . . . There can be no question of surrenderm"