Leaders of the three major nonwhite groupings in South Africa held an unprecedented meeting today and formed an alliance to establish a nonracial state.

Top representatives of the Zulus, the country's largest black tribe, Coloreds and Indians called for a convention in March to "formulate a common strategy against apartheid" and to draft a new constitution for South Africa.

The meeting marked the first time that political leaders have met across racial lines in South Africa at such a high level and represented a significant challenge to the white-minority ruled country. A union of 19 million blacks and 2.7 million Coloreds (persons of mixed race) has always been feared by the whites who have hoped that in any confrontation the Coloreds would side with the 4.3 million whites.

Any successful alliance would upset the government's plans to proceed with its policy of apartheid (racial separation), by setting up homelands for the blacks and separate parliaments for the other major racial groupings.

There is considerable question whether the government would allow any such alliance to come into existence since South African law prohibits multiracial political parties. It did allow the meeting to be held today although it apparently was inviolation of laws prohibiting members of one race from addressing a political party of another race.

After a hour meeting in the Zulu town of Ulundi, the representatives of the three racial groups announced that "the time is now right that the oppressed people of South Africa should come together to formulate a common strategy against apartheid.

The leaders of the nonwhite communities said they were willing to talk with whites "who want to be part of the human race in South Africa."

Attending today's meeting were Gatsha Butnelezi, chief of South Africa's largest racial group, the 5 million Zulus; Sonny Leon, president of the militant Colored organization, the Labor Party, and Y. S. Chinsamy, leader of the Reform Party, which represents Indian people in South Africa.There are about 750,000 Indians in the country.

The three parties said, the alliance would draw up plans for a constitutional convention" to which all South Africans will be invited" to discuss "a charter for a nonracial community and the new constitution for South Africa."

Under the constitutional proposals put forward by the ruling National Party there would be three parliaments -one each for whites, Indians and Coloreds. The government apparently hoped to draw the Coloreds and Indians into an alliance with the whites, but critics said that they would not be given any meaningful power in the new arrangement. The plan had been rejected by both the Labor Party and the Reform Party.

Buthelezi has so far rejected cooperation with the government plan to separate the black population into quasi-independent tribal homelands by ruling out such an idea for the Zulus.

Two years ago he revived a moribund Zulu organization, Inkatha, and began describing it as a liberation movement. Now, with 130,000 dues-paying members, it is the largest mass political organization among blacks in South Africa.

Before today's meeting began, Buthelezi told reporters, "What we have seen in the past few years is that the whites are not prepared to change. This has been demonstrated beyond doubt by the last elections - that nothing short of a miracle can make most whites of South Africa change on the basis of any rationality . . . The whites have arrogated to themselves to dictate to the rest of us the future which makes us chattel in the land of our birth."

The three parties at today's meeting all have extensive, though not total, support among their respective racial groups.

The government has tried to counter Buthelezi's drive for an alliance with warnings to the other racial groups. A government radio commentator, for instance, said that Inkatha is a "Zulu-inspired, Zulu-oriented and Zulu-led organization [that] has made many endeavors to extend its activity, and thus Zulu influence, by attracting to it members of other black nations."

"The prospect of being absorbed into a Zulu imporium," has sparked a desire among other blacks for even quicker independence in their respective homelands, the radio commentator said. Having thus been thwarted, he said, Buthelezi "now turns to brown men."

Only the afrikins paper Die Vaderland, made the frank admission as to why the Colored Labor Party is now joining Buthelezis Inkatha. "We ourselves must bear a great deal of the guilt for the fact that the brown people are moving away from us instead of closer. We have never made it exactly easy for them to draw closer."

In recent years South Africa's colored populations has identified more and more with the black population. This has been particularly ture of the group's youth who have held several sympathy demonstrations in support of various black causes.

Buthelezi initiated talks last year with the white Progressive Federal Party which has long been critical of the government's racial policies. According to one observer, "fundamental differences" have prevented Inkatha and the Progressives from getting any closer than occasional talks in a liaison committee.

It is too early to tell if today's meeting will give birth to a really viable, working alliance, and that if it does, whether the government will then allow it to live. If it does survive, its first major task will be to convince the whites that they are not "ganging up" on them.Given the present state of race relations in this country, that will be difficult.