U.S. Ambassador Herman Nickel, returning here three months after he was recalled to Washington in protest, called on the white-minority government today to move beyond "mere statements" and begin dismantling "key features" of its apartheid system of racial segregation.

"The injustices committed in the name of this system have gone on too long and they must stop," said Nickel in the strongest language he has used since he became ambassador in 1981.

A few hours before Nickel's arrival, President Pieter W. Botha, in a sharp escalation of rhetoric, accused the United States of initiating "a form of warfare" against South Africa with the limited economic sanctions announced by President Reagan yesterday.

Both men's remarks were some of the toughest yet exchanged by Botha's government and the Reagan administration, which has been considered South Africa's closest ally since black turmoil began here a year ago. The remarks set the stage for what could be an angry confrontation when Nickel delivers a personal message from Reagan to Botha later this week.

Botha, speaking briefly to a reporter for the state-run national television network, said South Africa had become the victim of an internal American dispute between Republicans and Democrats.

"South Africa is just the punch bag that gives them an excuse to vie with each other about who can impose the toughest measures," Botha said.

"America is busy with an immoral war in the name of morality. President Reagan's stand is disappointing because sanctions are after all nothing but a form of warfare," he added.

Later in prepared remarks to an audience at Rand Afrikaans University, Botha decried international "selective morality" that he said tolerates discrimination by nonwhites "against any group, white brown or black," but condemns the whites of South Africa.

He said South Africa's critics "often live in a dream world." He believed in Christianity, Botha said, "but nowhere does Christ teach me to commit suicide for the sake of my neighbor."

Botha's comments appeared to undermine previous efforts by South Africa to avoid direct criticism of President Reagan or his administration in response to the sanctions. In a statement issued last night, Botha had criticized the measures as "punitive," but had said he would withhold further comment until he had studied Reagan's personal message.

His comments also seemed to undermine statements by other South African officials today who said their government hopes to maintain relations with the United States and still considers viable the administration's policy of "constructive engagement."

"We want to have open and normal diplomatic relations with all the western countries, including the United States," said Deputy Foreign Minister Louis Nel in an interview. "Even Russia is not writing off the United States so how can we? America has the power to play a positive role."

Nickel, appearing at an airport press conference tonight, also said the administration was sticking with constructive engagement: "The president has made quite clear that we see no alternative." But he went on to make remarks far stronger than those he and other architects of the policy used to make.

Asked what South Africa could do to persuade the administration to lift the new sanctions, Nickel responded, "I think we have gotten beyond the point where mere statements or even just statements of intent are adequate. Things have to be seen to be happening. Negotiations have to be seen to be starting. Some of the key features of the apartheid system have to be seen to be abolished.

"I think that that is absolutely necessary to restore the international confidence that is so necessary to deal with any number of things, including, of course, this country's financial situation."

The ambassador said his message from Reagan "underlines our grave view of the current crisis and our assessment of what is needed to move from confrontation to negotiation here in South Africa and to restore confidence abroad."

He hinted that the United States might be prepared to play a mediatory role between the white government and its predominately black opposition.

"We understand that this is first and foremost the task of South Africans of all races," said Nickel, "but to the extent that we can and to the extent that the parties may find it useful, we are prepared to help."

Such a role would very likely be rejected by many leaders on both sides, with the government seeing it as an infringement on sovereignty while blacks would question the Reagan administration's impartiality.

Indirectly responding to Botha's charge that the sanctions would do economic damage to blacks and whites alike in South Africa, Nickel said Reagan's action was designed to make "a clear distinction between measures which would indiscriminately punish the people of South Africa . . . on the basis of an implausible theory that such suffering will somehow lead to a more just society and measures which are aimed at the machinery of apartheid."

He noted that many of the steps, including the ban on nuclear technology and on computer sales to the South African police and military, were already in existence. He omitted mention of other steps, however -- including a ban on loans to the government except for nondiscriminatory social welfare facilities and the enforcement of an equal-opportunity code on American businesses in South Africa -- that the administration had opposed until last week.

Nickel was recalled to Washington June 14 to protest a South African commando raid on suspected antigovernment resistance fighters in Botswana and an earlier abortive raid allegedly targeted at Gulf Oil installations in Angola.