The Colored (mixed-race) branch of South Africa's biggest and most influential church has voted to break relations with the white-only parent branch unless it renounces its support for the government's apartheid system of racial segregation.

Observers say the white church is unlikely to do this and that a traumatic breach between the two is certain. The Dutch Reformed is the largest faith in the country and the faction in conflict is the largest Dutch Reformed group.

This is likely to be followed by the church's African and Indian branches breaking away as well, resulting in the disintegration of the "family" of ethnically segregated Dutch Reformed churches.

There is also likely to be a split within the white church. Recently 136 white ministers put their names to an open letter condemning the church's support for apartheid.

Some of these are likely to quit if, as is expected, the church confirms its endorsement of the policy at its quadrennial synod due to start in Pretoria Tuesday.

Church sources say there will then be moves to merge the three nonwhite branches and the white dissidents into a single new multiracial Dutch Reformed church strongly opposed to apartheid.

These developments could have a significant influence on South African politics.

To begin with, the pending break by the Colored branch will deepen divisions between the ruling white Afrikaners and the predominantly Afrikaans-speaking Colored community at a time when Prime Minister Pieter W. Botha is trying to draw the two groups closer together.

A split by a substantial number of liberal white ministers would also deepen divisions within the once tightly knit Afrikaner community, whose unity and determination has enabled it to dominate the South African political scene for more than three decades.

Hard-line Afrikaners are rebelling against Botha's tentative overtures to the Coloreds and Asian minority groups, and substantial numbers are joining a new breakaway conservative party on the far right.

Any blow to the prestige of the Dutch Reformed Church will also have political implications. There are actually three offshoots of the Dutch Reformed Church brought by settlers to South Africa three centuries ago, and this one, the Nederduitse Gereformeerde Kerk, which is by far the biggest, is the moral arbiter of South African life.

Most of the government's Afrikaner supporters, and most members of the Cabinet, are among its 3.5 million members. While it does not intervene directly in politics, its moral influence on government is enormous and has been likened to that of the Roman Catholic Church in Ireland.

Its early declaration that apartheid was justifiable on theological grounds was thus of fundamental importance to the government -- and any attack on that justification, such as the Colored church is now making, is likewise important.

The looming upheaval in the church stems from a decision by the World Alliance of Reformed Churches, at a conference Aug. 25 in Ottawa, to declare apartheid a heresy and suspend the membership of the Nederduitse Gereformeerde Kerk and a smaller Dutch Reformed church called the Nederduitsch Hervormde Kerk.

The World Alliance, an international grouping of about 150 churches of the Reformed, Presbyterian and Congregational traditions, then elected a 37-year-old Colored South African, the Rev. Allan Boesak, as its president.

Boesak, an articulate and politically astute campaigner against apartheid, returned to South Africa to carry on his campaign through the Colored branch of the Nederduitse Gereformeerde Kerk, of which he is a member.

When its synod met in Cape Town last month, he made a run for the top position of moderator but the votes of white ministers assigned to the Colored branch as "missionaries" blocked him. He lost narrowly to a conservative Colored minister, Izak Mentor, who is committed to trying to retain links with the white church.

However, Boesak managed to gain the number two position of assessor, and an ally was elected actuary. That gave the anti-apartheid forces two of the four positions on the moderature, the church's controlling body.

As the synod progressed through last week the articulate Boesak gradually increased his influence over the 528 delegates. When it came to voting on the World Alliance's apartheid declaration Friday night, he won by a big majority.

The Colored synod passed a resolution declaring "that apartheid is a sin, that moral and theological justifications thereof make a mockery of the Gospel, and that its complete disobedience of the word of God is a theological heresy."