The Reagan administration, responding to mounting criticism and growing nonviolent protests, yesterday described as "rubbish" claims that the president's policies toward South Africa help promote that country's segregationist system.

The defense of President Reagan's position was made by Assistant Secretary of State Chester A. Crocker at an unusual White House briefing designed to convince reporters that Reagan policies of "constructive engagement" and "quiet diplomacy" are working.

White House officials said that the Crocker briefing had the dual purpose of responding to the protests while reminding South African leaders of U.S. distaste for apartheid.

But the protests continued unabated yesterday. Three more persons, including Rep. Parren J. Mitchell (D-Md.), were arrested at the South African Embassy in Washington, bringing to 19 the number who have been arrested there in nonviolent protests against U.S. policy. Four others were arrested at the South African consulate in New York.

Crocker claimed that "a process of change" is under way in South Africa and denied that the Reagan administration had given its blessing to apartheid.

"We have made very clear, and the president reiterated his concern this morning, our deep concern about moves of repression in South Africa that could shut down peaceful alternatives inside that country," Crocker said. "We have repeatedly made clear our concern about detentions, for example, of moderate black labor leaders . . . . "

It is rare for a State Department official to brief reporters on the record at the White House after meeting with the president. Officials acknowledged that the motivation for yesterday's session was partly to answer the nonviolent demonstrators who have put the national spotlight on the Reagan policy in South Africa.

Privately, however, a White House official acknowledged that these demonstrations might be "worthwhile in the long run" because they convey to the South African government the extent of U.S. distaste for the country's militantly segregationist system.

While Crocker did not mention it yesterday morning, he attended a dinner the night before where one of the guests was South African Bishop Desmond Tutu, winner of the 1984 Nobel Peace Prize. Tutu has charged that Reagan policies have made conditions worse for blacks in South Africa.

The administration is trying to take the sting out of the sit-ins, in which many of those arrested have been congressmen or celebrities, and the criticisms that have been made in recent days in well-publicized speeches by Tutu, Sen. Edward M. Kennedy (D-Mass.) and former Democratic presidential candidate Jesse L. Jackson.

Crocker did not try to defend recent repression in South Africa, saying that "we're extremely concerned about the repressive moves." But when he was asked to give evidence of his claims for positive peaceful change in South Africa, he cited "rates of spending for black education," the opening up of trade unions "on a democratic and peaceful basis to members of all racial groups" and the recent constitutional change that has been denounced by critics of the South African regime as a sham that perpetuates segregation.

The claim of progress within the trade unions is likely to have an ironic ring for critics of the Reagan administration's South African policies. One of their accusations, denied by Crocker, is that the United States has failed to speak up in behalf of a score of labor leaders recently detained in South Africa.

The administration's essential claim yesterday was that its policy has been misrepresented. Crocker said "a significant amount of pressure exists in our policy" and contended that "constructive engagement is not an embracing of any status quo."

Reagan's defense of his South African policies, reiterated last week in an interview with The Washington Times, is that it is producing "quite a bit of progress" toward an agreement that would link withdrawal of Cuban troops from Angola to a pullout of South African forces from Namibia.

Crocker was optimistic about an eventual agreement yesterday but gave no timetable for completing it.