The point of potential disaster for the Carter administration's bold initiatives in Rhodesia is clearly defined for Zbigniew Brzezinski.

It comes, in words President Carter's national security affaris adviser has reportedly used to other policy-makers, "when white nuns being raped by black guerillas wearing red stars on their armbands start appearing on nightly television news."

An intense and complex exercise of "positioning" is being conducted around Brzezinski's brightly colored epigram which some State Department planners see as the opening shot in a campaign that would lead to an eventual U.S. "disengagement" from Rhodesia and from the administration's early stress on building strong ties with African nationalists.

Brzezinski's aides deny this is his intent.He is instead portrayed as using his usual pithy, dramatic language to mobilize the administration into action that will head off the nightmare vision, and the potential superpower conflict that lies behind it.

In either case, this behind-the-scenes dialogue is beginning to shape the context of a key policy choice on Rhodesia that the Carter administration now expects to have to make in January. Almost inaudible behind the walls of Washington's bureaucracy, the December conversation is likely to have a major effect on the range of choices President Carter may face.

Far more is involved in this than the time-honoured Washington exercise of positioning, however. At stake is the fate of one of the brightest and boldest of the early Carter administration policy initiatives, one rooted in the president's own concern about civil rights at home. Also on the line will be some aspects of the special relationship that ties Washington to London.

Officially, Brzezinski, State Department specialists on Rhodesia and other authorities remain equally committed to heading off the disaster-in-the-making in Rhodesia by working with Britain to convene peace talks between the white-dominated Salisbury government and the Soviet-backed guerilla forces.

Privately, however, a number of them concede that the accelerating slide to disaster in Rhodesia has overtaken a number of the assumptions of the Anglo-American initative, and is forcing a high-level informal review of U.S. options.

Britain signaled its concern about the rapid deterioration last month by informally proposing to Washington s joint emergency evacuation plan to airlift out the roughly 165,000 British passport-holders and the 2,000 Americans now in Rhodesia, U.S. diplomats report. The State Department has thus far withheld agreement for such planning.

The trigger for new U.S. policy choices will be a report made this week to Prime Minister James Callaghan by Labor Party parliametarian Cledwyn Hughes, who has been investigating in southern Africa wheterh there are any chances now for convening an all-parties peace conference.

Callaghan will make public in January the Hughes report, which is widely expected to conclude that the Anglo-American approach still has not brought the sides to the point of being ready for talks.

The Callaghan government-which does not want to be left "holding the baby alone" months before an election-is expected to be able to do nothing more than "tread water" by insisting on continuing the Anglo-Americans approach.

But the Carter administration faces a different constellation of forces. Conservative Republican senators are moving to increase their forces on the Senate Foreign Relations Committee and to turn Rhodesia into the kind of major public issue that Brzezinski's nightmare image suggests is possible.

Three main trends are surfacing in the discussions surrounding a crisis that could affect world peace, the administration's policies and fortunes in Africa and perhaps careers. When Secretary of State Cyrus R. Vance begins to get option papers next month, they are likely to cover:

Disengagement. This would involve the United States tossing the Rhodesian problem back to the British to handle by themselves or, more likely, into the United Nations. This would cut U.S. diplomatic losses and hopefully minimize political damage at home in 1980.

Immobilization. The administration would go on repeating that the Anglo-American approach was alive and well while watching events produce a crisis that might be more susceptible to outside influences.

Safety net. The president would probably make a statement announcing the administration has done all it can to get the two sides together and is suspending its diplomatic activity until they commit themselves to talks.

The hope of this possible option, which has strong support at the State Department, would be that the announcement would sober the internal government of Prime Minster Ian Smith and the guerilla forces of Joshua Nkomo and Robert Mugabe enough to get them all to the peace table.

"The choice is likely to boil down to one of being more active or less active," one administration planner said. "Nobody is openly backing total deisengagement now, although there is some groundbreaking going on."

Any open attempt to argue the disengagement course would run headon into British opposition. The British could issue public reminders of the joint paternity of the Anglo-American peace plan, which Carter personally approved, point by point, in a White House session in July 1977.

Moreover, Callaghan called Carter before announcing the Hughes mission last month and got Carter's agreement to a public announcement of U.S. support for the mission.

While reportedly not advocating disengagement, Brzezinski is known to be increasingly convinced that a moderate western-sponsored solution cannot be produced by negotiations as long as the Soviet Union and Cuba continue to provide arms and training to the guerillas.

If these pipelines cannot be shut off by diplomatic action, Brzezinski reportedly fears that the administration could be forced by domestic pressures into having to choose between backing the Smith regime-which he finds unacceptable-or having to abandon its Rhodesia diplomacy altogether.

He and other officials reportedly feel that the administration's January choice, even if it is to stay on the present course, will have to be coupled with a major campaign at home to explain more clearly the Rhodesian policy to the public.