The State Department, in a belated quest for greater support of its controversial South Africa policy, has set up a public relations bureau known as the "Working Group on South Africa and Southern Africa."

In charge of the effort is a former member of the White House Office of Public Liaison, Douglas Holladay, who is the first to admit that his task is difficult. "There is a lot to be done," he said, to explain to the American public why the administration has so vehemently opposed economic sanctions against South Africa as well as calls for American firms to leave.

"People didn't understand what we were doing and why we were doing it," he said in an interview Friday. The House and Senate "didn't feel on board" the administration's policy-making process, he added, referring to the overwhelming vote in Congress last year in favor of U.S. sanctions against South Africa -- a vote that repudiated the administration's policy of "constructive engagement."

While the administration's official policy toward South Africa is still "constructive engagement," a euphemism for quiet diplomacy, there is a marked tendency these days for the Working Group's members to avoid any use of the term and to talk instead about other things.

The history of the Working Group began in August. Just a few weeks before President Reagan reversed himself under enormous congressional pressure and announced limited economic sanctions against South Africa, Secretary of State George P. Shultz temporarily recalled U.S. Ambassador to Zimbabwe David Miller and assigned him to setting up the office.

The original model for the Working Group on South Africa was the department's Office of Public Diplomacy for Latin America and the Caribbean, set up in July 1983 "to educate the public" about the Reagan administration's Latin America policy.

While the Working Group is still being organized, the thrust of its programs and efforts may be different from those of the Latin America office if Holladay's present thinking prevails.

Named permanent director only in December, Holladay is a former professor at the University of Virginia with a degree in theology from Princeton University and one from Oxford University in 19th century British social and political history. Before serving in the White House, he was associate deputy undersecretary in the Department of Education for two years.

Almost immediately after being named director, Holladay left for South Africa on a visit that seems to have left a deep impression on him and perhaps transformed his ideas of what the Group ought to be doing.

He now seems far more interested in finding new ways to engage American energies in the struggle for change in South Africa than in mounting a big public relations campaign inside the United States.

"There is an enormous amount of frustration in this country over South Africa, looking for outlets," he said, referring to campaigns to pressure American companies to leave South Africa and to impose U.S. sanctions. Holladay strongly disapproves of such tactics, saying, "We have got to find other ways to express that frustration."

One of his ideas is to use the Working Group as a liaison between private American groups and individuals wanting to help improve conditions of impoverished South African blacks, and groups and institutions needing outside aid. For example, he suggested that American students, educators and others might work during the summer in black townships of South Africa. The Peace Corps does not place volunteers in South Africa.

Another role Holladay sees for his office is to act as "a catalyst" for more meetings and discussions between blacks and whites inside South Africa. The United States, he said, might be able to help bridge the gap between the South African government and the country's black population.

"The United States has a legitimate catalyst role in helping to get people to talk to each other in that country," he said. "Everyone I talked to there wants some way out of this complex, contradictory situation. But they did not know what to do, how to move."

Holladay said he also intends to see his office play a role in the formulation of U.S. policy toward South Africa and the rest of southern Africa. Like most of his colleagues, he avoided using the phrase "constructive engagement," saying "I don't hear a lot of people using the word any more." But, he added quickly, "We feel it is better to be involved, engaged in South Africa."

Holladay's first big challenge will come if -- as seems likely -- the administration decides to become involved in the Angolan conflict on the side of the South African-backed opposition led by Jonas Savimbi, who is waging a guerrilla war against the Marxist central government. Such a decision is certain to be seen by South African blacks as a sign that Washington is lining up with the country's white leaders, and would surely complicate Holladay's plans to have the Working Group act as an intermediary in the internal conflict. When questioned on this issue, Holladay was noncommittal but appeared clearly troubled.