More than a score of the 142 members of the ruling National Party voted today at a caucus of its members of Parliament against a motion of confidence in the leadership of Prime Minister Pieter W. Botha, bringing the party to the verge of its biggest split since it came to power in 1948.

The quarrel has grown out of differences between Botha and conservatives within the party over whether the nation's 2.5 million Coloreds (persons of mixed race) should have some form of representation in the presently all-white Parliament.

Following the confidence vote at the party's parliamentary caucus in Cape Town, Botha said he had given the 22 members who voted against him until next Wednesday to reconsider their positions, failing which "disciplinary action" would be taken. That would almost certainly mean expulsion from the party, leaving the rebels to go into opposition either as independents or as a new right-wing party.

The only other significant split since the National Party came to power was in 1969, when the then-prime minister, John Vorster, expelled four members of Parliament for opposing his relaxation of the party's apartheid, or strict segregationist, policies to allow some racial mixing in sports.

They formed the right-wing Herstigte Nasionale Party, which made significant gains in a general election last year but still has no parliamentary seats. Today the leader of the party, Jaap Marais, said he believed the rebels would eventually join him after a period of reflection.

Fredrik Van Zyl Slabbert, leader of the liberal Progressive Federal Party, which has 27 seats in Parliament, said, ". . . the unity of the National Party is finally destroyed. We are now entering a new process that will lead to a realignment, which is essential if there is to be reform in South Africa."

Proposals recommending that Coloreds be given some type of parliamentary representation are expected to come from a constitutional committee of the President's Council, an advisory body, in May. Earlier in the week Botha indicated support for the idea, which has been opposed by the party's leading conservative, Andries Treurnicht, the minister of state administration.

Treurnicht wants Coloreds to be confined to a separate parliamentary chamber, with only advisory rights, the white chamber retaining exclusive legislative power.

On Monday Botha told parliament: "There can logically only be one government in a country." This triggered the intraparty dispute.

Today Botha called a caucus to thrash out the difference, ending it with a call for a vote of confidence in his leadership and his interpretation of party policy. One hundred members voted for him, 22 against; there was one abstention and 18 caucus members were not present. Botha said afterward that Treurnicht had walked out without voting.

A split of this size, if it becomes permanent, would still leave Botha with a safe majority as prime minister, but the political repercussions would be far-reaching and the effect on the dominant Afrikaner element of the white minority traumatic.

Afrikaner nationalists, feeling themselves to be an endangered "nation" in a continent where they are heavily outnumbered by blacks, have always given priority to the maintenance of unity, often papering over deep differences to avoid splitting.

If the split occurs next Wednesday, the most immediate possibility is that it could quickly widen into a revolt by the Transvaal branch of the party, which Treurnicht leads and which has half the party's 142 parliamentary seats in the 177-seat chamber.

The Transvaal is the heartland of conservative resistance to the cautiously reformist line Botha has taken since becoming prime minister in 1978. All but two of the rebels are from Transvaal.

Which way Treurnicht eventually goes will be crucial. In a statement tonight he indicated that without modification of Botha's views he could not go along with what he regarded as a longstanding party position against "power sharing" with nonwhites.

He will probably give a clearer indication of his intentions on Saturday at a meeting of the Transvaal branch's 16-member leadership committee.

If he joins the rebels, his most likely tactic would be to call an emergency congress of his Transvaal branch. Since the National Party is federally structured, with each branch having autonomy, he could demand a vote of confidence, just as Botha did.

Assuming that he won, which is likely, he could expel dissenters and take the entire branch into opposition against the parent party.

It is this possibility that has caused Botha in previous years to shelve reformist plans rather than confront Treurnicht openly. But tensions between the party's reformist and conservative wings has been building up for years and has now apparently reached the breaking point for Botha.

Most political analysts interviewed today thought the split was inevitable and felt that, while it might cause Botha to become even more cautious in the short term, in the longer run it could lead to a major political realignment that would open the way to reforms.