Secretary of State Cyrus Vance ended his talks here today on notice that the Anglo-American plan for Rhodesia is dead and that the United States must find a substitute approach to what Vance called the "new reality" of the explosive Rhodesian situation.

That was the outcome of Vance's three days of discussions with Britain's new Conservative prime minister, Marvaret Thatcher, and her foreign secretary, Lord Carrington.

Although everyone involved characterized the talks as friendly, they still ended with the British resolved to pull out of the plan that has been the cornerstone of the Carter administration's policy on the racial conflicts of southern Africa.

Under that plan, forged in cooperation with the former Labor government here, Washington and London had refused to deal with Rhodesia's white-dominated transitional government, while insisting on a solution that would include black guerrilla forces fighting the Salisbury government.

"I would have to say that there is a new reality," Vance said following his final meeting with Carrington. "We must recognize that there is."

His words immediately triggered speculation that the United States plans to follow Britain in moving toward recognizing the recently elected black-majority Rhodesian government to be headed by Prime Minister-elect Abel Muzorewa.

U.S. sources in Vance's party were quick to insist, however, that his remarks were misinterpreted. They said Vance only was referring to what one called "the new situation on the ground there." This is the fact that the Salisbury government, despite intense disruptive efforts by the guerrillas, held elections that produced a black-majority government that President Carter will have to take into account in deciding what to do next about Rhodesia.

Under legislation passed by Congress last year, Carter must determine whether the elections were sufficiently free and fair to warrant lifting U.S. economic sanctions against Rhodesia and moving toward cooperating with Muzorewa. Congressional pressure to take that route has built up to the point where the Senate last week called overwhelmingly for an immediate end to the sanctions.

Now the Thatcher government's withdrawal from the Anglo-American plan seems likely to intensify the pressure.Until now, the administration had been able to justify its Rhodesia policy, in part, by arguing that Rhodesia, as a breakaway British colony, was primarily a British problem and that the United States was cooperating with Britain to handle the situation.

Carter, who is expected to make a decision on sanctions in June, now must decide whether conditions justify continued U.S. shunning of the "internal settlement" that produced the Muzorewa government. Up to now, his policy has been based on a belief that any lasting solution to Rhodesia's strife must include the guerrillas and on a fear of antagonizing black African countries that support the guerrillas.

These are still questions of great concern to U.S. policymakers. But the president now must weigh them against the possibility that Britain's change of direction could snowball into a rush of other Western countries recognizing Rhodesia-a situation that would leave the United States increasingly isolated and put Carter in danger of being overruled by Congress.

Yet, while the president obviously is in a tight spot, U.S. officials with Vance insisted that his talks with Carrington had gone well and that the British decision had not caused any acrimony between the two governments.

These sources explained that Vance knew in advance that the government here was bound by Thatcher's campaign promises to move toward recognition. As a result, the sources said, Vance came not with any idea of trying to talk Carrington into sticking with the Anglo-American plan, but to make the breakup as harmonious as possible.

In that respect, the sources insisted, the talks went well. Carrington, who publicly announced the British policy Tuesday, promised to mve toward recognition in slow stages that will minimize the immediate pressures on Carter. And Vance, in a bow to Carrington, denied a British press report today that the United States had argued strongly against the Thatcher government's course.

British sources said London was lightly disappointed that the two were unable to find an immediate basis for a new joint approach to Rhodesia. Instead, they said, Vance insisted throughout the talks that Carter has not yet made the decisions required by Congress and that, until he does, the United States is unable to predict its future approach to Rhodesia. CAPTION: Picture, Secretary of State Vance is greeted by Prime Minister Margaret Thatcher during his trip to Britian. AP