After months of agonizing, the major political party among South Africa's minority Coloreds, as people of mixed race are called here, voted tonight to participate in a new constitution that will give them a token role in the country's government.

It was an important breakthrough for Prime Minister Pieter W. Botha, who split his white supremacist Afrikaner National Party when he advocated the reform last year and who risked a serious political blow if the Coloreds rejected the plan. At the same time it was a setback for Chief Gatsha Buthelezi, the most important black leader allowed to operate in South Africa. He had urged the Coloreds to reject the new deal because its reforms affected only them and the small Indian minority while excluding the black majority.

Buthelezi indicated he would regard acceptance as an alliance of whites, Coloreds and Indians, who together number about 8 million, against the 21 million blacks, and he predicted a violent reaction.

The Colored Labor Party, torn between wanting a foothold in the white-only political system and not wanting to break away from the black majority that party members believe will rule South Africa one day, settled for an uneasy compromise. The annual congress approved a resolution rejecting the principle of the new constitution but agreeing to participate in the process in the hope that it will give Coloreds a political platform from which to negotiate for further reforms.

The 500 delegates appeared to swing back and forth throughout the meeting as speakers for both sides addressed the group.

They cheered Buthelezi yesterday when he delivered the opening address and told them it could be their "year of glory" if they rejected the proposals.

But they swung back this morning when it became apparent that their party leader, Allan J. Hendrickse, a Congregationalist minister who had avoided taking a position on the new constitution after Botha proposed it in mid-1982, was in fact eager to participate.

By nightfall, the party's national chairman, David Curry, clinched matters with a rousing speech in which he said, "It is politically expedient to use this opportunity to get what we want. South Africa is irreversibly in a process of reform. We must get in there and make ourselves part of the process."

Although many party members abstained, only 13 delegates voted against the resolution. Several said later that they would resign from the party, and a strong reaction is expected from young members who have been advocating a closer alliance with the voteless blacks.

The real loser was Buthelezi, who has just gained increased political prestige by successfully resisting a government attempt to cede two areas of tribal territory to the neighboring kingdom of Swaziland, which would have deprived another 1 million blacks of their South African citizenship.

This defeat will offset that gain, another matter of satisfaction for Botha although some observers think it may mean increased black support for the outlawed African National Congress.

Buthelezi is committed to resisting the government's policy of drawing the Colored and Indian minorities into the white political system, while turning all blacks into foreigners without a claim to political rights by making them citizens of tiny tribal "homelands," which are given nominal independence.

He has partly thwarted the government's plan by refusing to accept independence for his own "homeland" of Kwazulu, the home of South Africa's biggest tribe, the 6 million Zulus. By thwarting the homeland drive, the Zulus have so far maintained their South African citizenship.

He also has tried to use his position as chairman of a coalition of nonwhite parties, called the Black Alliance--which includes the Colored Labor Party and the small Indian Reform Party--to persuade his minority group allies not to be co-opted into the white system, as he puts it.

It was decided before Botha announced his constitutional plan to hold this year's Labor Party Congress in Kwazulu and to invite Buthelezi to open it. This gave Buthelezi an opportunity yesterday to make a strong pitch to the delegates, and the fact that they ignored him on his own home ground makes his defeat all the more bitter.

"You have to make your own decision," Buthelezi told the convention, "but it may be helpful to you to know how Africans feel about it.

"For the Colored community to accept proposals which exclude their African compatriots would be a disaster for them and for everybody. It would be interpreted, rightly or wrongly, as your ultimate abandonment of Africans as your fellow countrymen and fellow blacks."

It could add a violent black-versus-Colored dimension to the growing confrontation between blacks and whites, Buthelezi said.

"The Colored and Indian people will become, in our eyes, second-class 'enemies,' " Buthelezi said. "One sometimes respects one's large enemy but seldom his little runners and camp-followers. They are without honor at all."

Colored leader Hendrickse scarcely referred to Buthelezi's appeal in his address today. He spoke of the Coloreds seizing an opportunity to play a conciliatory role between black and white.

"The time for protest politics is passed," said Hendrickse. "We must go in and speak to the government from a platform to which they will have to listen."

Quoting U.S. civil rights leader Martin Luther King Jr. he said, "You must not wait until the day of full emancipation before you make a creative contribution to the life of the nation."

Botha's proposals are to establish a parliament that has separate chambers for whites, Coloreds and Indians and has an executive president who in practice will always be white.

The white chamber will be dominant, and if either the Colored or Indian chambers obstruct any legislation it will be passed on to a fourth super-chamber, called the President's Council, where the white-controlled government will have an assured majority. The decision of the President's Council will be final.

The Indian Reform Party, the principal opposition party within the South African Indian Council, is due to consider the constitutional proposals later this month. Its leader, Yassan Chinsamy, said today he thought it would reject participation.