South Africa's main Dutch Reformed Church, which is so close to the governing party that it is jokingly referred to as "the national party at prayer," has failed to accede to demands that it withdraw its theological approval of the government's policy of apartheid or strict racial segregation.

The demands came from the World Alliance of Reformed Churches, the main ecumenical body to which the South African church belongs, and from its own segregated branch for mixed-race, or "Colored," people which threatened to break off relations unless the demand was met.

Observers believe a breakaway by the Colored branch is now inevitable.

The all-white church founded here by Dutch settlers three centuries ago did not specifically reject the demands at its synod which ended here Saturday. It avoided taking decisions on these and other key racial issues by referring them to standing commissions that will not report again until the church's next synod four years from now.

The World Alliance, an international grouping of about 150 churches of the Reformed, Presbyterian and Congregational traditions, decided at a conference in Ottawa in August to declare apartheid a heresy and call on the South African church to revoke its approval of the policy. It suspended the church's membership pending a decision.

The World Alliance also required the South African church to end its insistence on segregated worship. The church here is divided into a main branch for whites and separate, or "daughter," branches for African, Colored and East Indian members.

Last month the church's Colored branch decided at its own synod to endorse the categorizing of apartheid as a heresy and to send an ultimatum to the main South African church. This warned that it would break off relations unless the church repented for having provided the "moral and theological grounding" for the government's policy.

At the same time 123 of the South African church's white ministers and lay members -- later joined by 15 others -- wrote an open letter to the church's governing body, calling on it to reject apartheid as un-Christian. Although this group is only a small minority within the church, its letter was widely publicized and caused a considerable stir in church and political circles.

These challenges confronted the church with the forbidding twin prospects of total isolation internationally and a major split within its own family of churches.

Yet from the moment its synod began here Oct. 12, it was apparent the church would strive to shelve the issues rather than face them squarely, out of fear of splitting its own white Afrikaner membership.

This is because the once tight-knit Afrikaner community is deeply split over politics, and the church does not want to add to the disunity.

A government plan to change the South African constitution by extending limited political rights to the Colored and Indian minorities, but not to the voteless black majority, has caused arch-conservatives to break away and form a new Conservative Party that shows signs of significant growth.

A smaller, more conservative Dutch Reformed Church, which was suspended at the same time by the World Alliance, has already decided to resign its membership rather than respond to the alliance's demand.

Many members of the main church wanted to do the same, and the moderature, or governing body, clearly feared that if the synod showed any sign of being soft in its response to the World Alliance they might quit and join the more conservative church.

On the other hand, a forthright rejection of these criticisms of apartheid might provoke some of the more liberal members into quitting and perhaps moving closer politically to the opposition Progressive Federal Party which advocates racial integration.

The synod therefore agreed to refer the question of apartheid to a commission that will "update" the church's official race policy and report to the next synod in 1986.

The synod then voted by a majority of four of its 464 delegates to resign from the World Alliance, short of the two-thirds majority required to take such a step. The moderator counselled another four-year postponement.

The synod sent a message to the Colored branch promising to go back to the Bible to search for answers to "the various stumbling blocks" in the relationship between them, and urged the other church to do likewise.

The synod passed a resolution rejecting racism but decided this could not be equated with apartheid. It rejected another resolution urging the government to rescind apartheid laws which forbid interracial marriage and sex.