The Anglo-American plan for a settlement of the Rhodesian dispute received another setback here today as South African Prime Minister John Vorster reportedly made clear his strong disagreement with its provisions regarding the security force both before and after the advent of black-majority rule there.

After nearly six hours of talks between Vorster and British Foreign Secretary David Owen and U.S. Ambassador to the United Naitons Andrew Young, both sides refused even to characterize what were privately reported to have been tough and difficult discussions.

"We've really got nothing to say," Owen said as the Anglo-American delegation emerged from the conference hall where they had been closeted with Vorster for most of the day. "These have been detailed discussions and we've got to go now to talk to other interested parties," he said.

"We've fully discussed the Rhodesian issue and I'm not in a position to characterize the talks in any way," South African Foreign Minister Pik Botha remarked tersely.

South Africa's position on the proposals for a Rhodesian settlement is crucial because as the latter country's only trading partner it can apply considerable economic pressure.

Despite the official reticence, informed sources close to the Anglo-American delegation described the future of the joint plan for a peaceful resolution of the 12-year-old Rhodesia dispute as "tenuous."

They said that Owen had been virtually "interrogated" by Vorster on various aspects of the proposed security force.

It was understood that Vorster was extremely critical of a provision calling for the virtual dismantling of the white Rhodesian army to make way for an African-dominated one.

Indications were that he was proving just as stubborn on this point as the five African "front-line" leaders had on provisions reportedly calling for the partial disbanding of the black nationalist guerrilla army. Their resistance on this surfaced in talks over the weekend with Owen and Young in Lusaka, Zambia.

It thus appears that the Anglo-American plan risks strong opposition from both the white and the black nationalist leaders of Rhodesia, as well as from their respective key supporters.

But all parties involved - the four black nationalist factions, the white-minority Rhodesia government, the five "front-line" African states, Sout Africa and the United States and Britain - have apparently agreed to refrain from stating their positions formally until the plan is published Thursday. That is the day after Rhodesia holds an election in which the current government of Prime Minister Ian Smith is being challenged by a conservative faction.

Nonetheless, the judge from the few comments aides have let drop and the expressions of the British and American diplomats involved in the complex negotiating process, things did not go very well at all in the two sessions of talks here today.

Both Owen and Young rushed away both times to avoid answering the barrage of questions from reporters.

Before the morning meeting, police removed posters from the conference building that said "Young is Our Enemy, Kick Him Out." The signs appeared in various points around the city as well.

But this was the only sign of anti-Americanism apparent today, and the South African state radio even praised Young for his speech in Lagos, Nigeria, last week opposing the imposition of sanctions on South Africa.

According to Prime Minister Smith, who met here Saturday with Vorster, the South Africans gave him "their blessings and hopes for a settlement."

"Whether it is an internal settlement or an external settlement, the South Africans leave it to our best judgement," Smith was quoted as telling a reporter.

Reports from Salisbury, the Rhodesian capital, say that Smith has been exuding a new confidence in public lately. The reports link his new mood to probable assurances of South African backing for his bid to pull off an internal settlement with moderate black nationalist leaders inside Rhodesia. Such a solution would totally bypass the Patriotic Front, the umbrella grouping of more radical nationalists and guerrillas that has the support of the five "front-line" African states.

Observes here believe that South African doubts about the Anglo-American proposal for first a neutral international force and then the immediate establishment of a black-dominated army have led Vorster to hamore sympathy for the Smith plan than he previously had shown.