President Reagan escalated his rhetoric against South Africa's apartheid system of racial segregation yesterday, calling on the Pretoria government to "reach out to its black majority" and urging the nation to "move toward a more just society."

In a speech marking International Human Rights Day, Reagan spoke more forcefully than he has in the past on South Africa, saying that "we feel a moral responsibility to speak out on this matter, to emphasize our concerns and our grief over the human and spiritual costs of apartheid."

Reagan's policy of "constructive engagement" with the Pretoria government, seeking to work through diplomatic channels rather than imposing sanctions, has been the target of intensifying protests and criticism in the United States in recent weeks.

The president was urged by South African Bishop Desmond M. Tutu last Friday to call for an end to the current violence in South Africa, the release of all detainees, a lift on bannings and forced population-removal schemes, amnesty for all political prisoners and a national convention to draw up a blueprint for a new kind of society.

Reagan previously has insisted that "quiet diplomacy" was the best means for bringing about change in South Africa, but officials said yesterday that he was now turning toward public pressure. The president made apartheid a major focus of his 21-minute address on international human rights and appeared to incorporate some of Tutu's suggestions. Reagan did not go so far as to endorse economic sanctions against South Africa, however, as some critics have demanded.

A senior administration official who briefed reporters on the speech said Reagan was speaking out because of a recent increase in repression in South Africa and because of the apparent view among Americans that the quiet-diplomacy approach was not getting results.

Reagan spoke before a group that included refugees from the Soviet Union, Iran, Nicaragua, Cuba, Afghanistan, Cambodia and Poland, as well as family members of people persecuted in those countries.

Reagan said that "quiet diplomacy has brought about humane and democratic change" in some nations that have authoritarian governments and enjoy "friendly ties to the United States and the community of democratic nations."

The president added that sometimes "quiet diplomacy is not enough . . . . "

Noting that the United States has declared that it views "racism with repugnance," Reagan said the administration wants "the government of South Africa to reach out to its black majority by ending the forced removal of blacks from their communities and the detention, without trial, and lengthy imprisonment of black leaders."

"Such action can comfort only those whose vision of South Africa's future is one of polarization, violence and the final extinction of any hope for peaceful democratic government," Reagan said. "At the same time, we note with satisfaction that the South African government has released 11 black leaders, including the top leaders of two of that country's most important labor unions."

The president added that "we ask that the constructive changes of recent years be broadened to address the aspirations of all South Africans." He said peaceful change in South Africa and in the region "can come only when blacks and whites find a durable basis to live together."

He also vowed that the United States would provide "unswerving support" if South Africans "address the imperatives of constructive change."