The United States and Britain received a qualified measure of support from South Africa last night for a new conference to try to keep afloat the Anglo-American plan for peace in Rhodesia.

"We can't force the parties together," South African Foreign Minister Pik Botha told reporters after conferring with Secretary of State Cyrus Vance and British Foreign Secretary David Owen, who carry their disputed plan to Salisbury, Rhodesia, today. But South Africa, Botha said, "will work toward creating the kind of atmosphere amenable to reaching such a solution."

There is "a different ball game" in Rhodesia since last month's creation of an interim government headed by white Prime Minister Ian Smith and three moderate black leaders, Botha said.

South Africa cannot "knock them over the head to coerce them to accept things they do not want to accept voluntarily," he said.

In fact, South Africa has tremendous leverage over its close ally which, like this country, has been ruled by a small minority of whites. South Africa controls the major access to land-locked Rhodesia and its economic and military lifelines. What South Africa fears most is that its Rhodesian neighbor will fall under the control of militant Marxist black nationalists, who in turn would intensify the struggle for majority rule in this nation.

As a consequence, South Africa is expected to follow a wary course of encouraging Rhodesia's new leadership to bargain for an end to the Rhodesian guerrilla war, but not at the cost of surrendering power to the most militant black leaders.

Rhodesia's new leadership last week rejected the Anglo-American call for a conference with the Patriotic Front, the guerrilla alliance which is fighting for the immediate removal of Smith. This would occur under the Anglo-American plan, but the Patriotic Front, in a conference in Tanzania that ended Saturday night, caught Vance and Owen by surprise with new, stiffer demands under the Anglo-American formula for black majority rule.

Vance, who is making his first official visit to South Africa, and Owen therefore are now attempting to sustain the negotiating process under greater obstacles than they faced just a few days ago.

The two foreign secretaries read a measure of hopefulness into what they heard from Botha last night on the Rhodesian dilemma and also on negotiations for granting (Southwest Africa), which is governed by South Africa.

Vance left Washington expecting that the Namibian problem would prove far more tractable than the Rhodesian conflict. A Western plan formulated more than a year ago by five members of the U.N. Security Council is now up for acceptance or rejection by both South Africa and the Southwest Africa Peoples' Organization (SWAPO), which is conducting a guerrilla struggle against South African rule.

"We would hope all of us working together can find a solution to both the Namibia and Rhodesian problems," Vance told a news conference last night.

The South Africans, Vance said, now have raised "some important issues" which will require further clarification.

Botha told reporters that the South African government "did not understand" the portion of the plan which requires that a residual force of 1,500 South African troops withdraw from Namibia within one week after the elections for a constitutional assembly. A United Nations peacekeeping force would be sent into the territory to supervise the transition from white minority to black majority rule.

Botha also indicated that additional points require clarification and said that the plan itself is still under study by organizations in Namibia.

Sam Nujoma, president of SWAPO, met with Vance in Dar Es Salaam, Tanzania, early this morning, and olso raised a number of questions about the proposal, American sources said. Nujoma expressed concern that the size of the U.N. forces are not specified in the Western plan.

Meanwhile, the Anglo-American plan for a Rhodesia settlement apparently foundered in discord Saturday as the co-leaders of the Patriotic Front, Joshua Nkomo and Robert Mugabe, raised demands for some fundamental changes.

Nkomo, returning from the talks with Vance and Owen to his headquarters at Lusaka, Zambia, told reporters that the two Western powers had not been as forthcoming as the nationalists.

Washington Post correspondent David Ottaway reported from Lusaka that Nkomo said the nationalists have made "concessions" regarding the executive powers of the proposed British commissioner on police and defesne matters.

"We moved toward them as far as we could but this was not reciprocated," he said.

Finding a compromise between the British and American proposals and the Front's stand had been as difficult as mixing water and oil, he added.

But Nkomo said he and Mugabe had agreed to attend an all-parties conference "with the understanding that it is a conference based on the Anglo-American proposals and a constitutional conference." He said Vance and Owen had talked of holding it ins late May rather than April 25, as was first proposed, bu that the two Western envoys had been vague about the exact date.

Nkomo said the two sides had agreed on a U.N. military and police force to serve in Rhodesia during the transition period but said the nationalists had insisted on a "strict mandate" for such a peacekeeping body. At no time, he said, could it be used to suppress the "struggling masses of Zimbabwe," the nationalist name for Rhodesia.

He said, however, that no agreement was reached on the exact composition of military forces under the transitional government. He said the Front wanted to see its guerrillas patrolling the country together with U.N. forces and "acceptable elements" of the Rhodesian army in a tripartite force, but he mode it clear this issue was still far from being resolved.

There was also apparently no agreement reached with Owen and Vance on the question of who is to supervise the elections for a new majority government. The Anglo-American proposals say the British resident commissioner is to be responsible but Nkomo said it should be the United Nations and "nobody else."