A two-week school boycott by colored (mixed race students) escalated into a confrontation with the South African government today as riot police broke into a locked meeting here and arrested 645 striking students.

Later, Prime Minister Pieter W. Botha vowed he would oppose "attempts to push this government around."

As in 1976, when school strikes set off 18 months of violent upheaval in black communities that left 575 people dead, the current challenge -- to the white minority government's avowed commitment to reform -- comes from students.

Hundreds of classrooms across the country at secondary schools and universities assigned to the colored population have been silent and emptied of students, who are demanding one integrated national education system and an end to what they call an "inferior, racial" system.

The protest, which is estimated by colored leaders to involve upward of 100,000 students and has the backing of colored teachers, parents and political leaders, poses a delicate problem for the government. It involves the 2.5 million coloreds, descendants of the original white European settlers and the indigenous blacks.

The government considers the coloreds to be its closest allies in its effort to set up new constitutional arrangements that will share a measure of political power, yet skirt black majority rule.

An unsatisfactory resolution of the boycott, which today's developments appeared to portend, could promote the coloreds' already growing identification with the black majority as they become more alienated from the government.

Police dressed in riot gear today shed the low profile posture they had taken toward the boycott and jumped over a locked school gate at a colored secondary school in Johannesburg, where an estimated 2,000 students were singing songs at a protest meeting.

The police ordered them to disperse from the school grounds, but before all could get out of the gate, 645 were arrested and later charged, with attending an illegal gathering. Police said they made the arrests because stones were thrown at them, but several eyewitnesses said no stones were thrown. Others cited incidents of truncheon-wielding police brutality.

Later in the day the most aggressive government reaction to the boycott so far came from Prime Minister Botha. Speaking in Parliament, Botha said the government would oppose "attempts now being made outside Parliament to push this government around from behind the uniforms of schoolchildren. I want to warn those people they are going to get hurt and if there are some unfortunate consequences they must not come with recriminations later."

"The government knows who the troublemakers were. A few have already been detained," Botha said, as he urged parents to persuade their children to go back to school. Police have arrested possibly as many as 30 black and colored leaders since the boycott began.

Then, seeming to add insult to injury, Botha announced that though he intends to "accommodate" coloreds and Indians in a new constitutional set-up, he is not in favor of a common voters' roll with thse two groups. This is tantamount to a rejection of the colored community's demands for equal citizenship with whites, which has underpinned the students' boycott demands.

Today's police action angered colored parents and teachers. "Today the police showed they are the real agitators," said Yvonne Petersen, a colored secretary at the raided school.

"I brought up my children as Christians and we stand wholeheartedly behind them, but we insisted they do it in a peaceful way and they were doing it peacefully," said George Thomas, a glazier earning $108 a week who was at the police station to get three of his arrested sons. "I'm not happy about this. We have been put down too long."

Only a few weeks ago the government rescinded the coloreds' top legislative body, the Colored Representative Council, because its elected members were not cooperating with the government. A wholly appointed body is to take its place.

This action has been cited by many analysts as a reason for the widespread support parents and teachers are giving the striking students.

"It's definitely different from 1976. Then it took the parents much longer to give their support and then it only happened in the midst of incredible violence," said colored university chaplain Allan Boesak, "But now the parents know right away that the physical condition of the schools is a result of the fact that the so-called 'colored' community is regarded as second-class citizens."

Typical of the colored community's reaction to the boycott is that of the Colored Teacher's Association of the Transvaal Province. "Our children are only demanding their fair share of the wealth of South Africa . . . We will not be alienated from them," its executive committee said in a statement. Similar responses have come from leaders of the all-colored Labor Party.

Part of the explanation for this solid community support has been the student-initiated meetings with parents to explain their actions. But expanding pressures for reform and the experience of 1976 have also contributed.

Boesack recalled the words of one colored factory worker at one of the student-parent meetings: "I don't know much about politics but I do know my children are right and I do know that the only thing we need is coresponsibility in the central government."

"There's no doubt that the outcome of this will affect the relationship between coloreds and whites," said Boesack. "Our traditional sense of belonging with the whites was shattered in 1976 and since then the gap has widened. This time, it's far more serious [than in 1976], there's no such thing as trust in the government and its promises."

Under the present separate educational systems, the government spends about four to five times as much percapita on white students as it does on coloreds, and about 10 times as much on whites as on blacks.

South Africa's population consists of 20 million blacks, 5 million whites and 2.4 million coloreds.