Some senior administration officials were deeply disappointed this week by South African President Pieter W. Botha's failure to announce significant reforms in his country's system of apartheid, or rigid racial segregation. But while his aides expressed dismay, President Reagan himself "remains optimistic" and unshaken in his policy of trying to promote reform through persuasion, not coercion, according to White House officials.

When his top foreign policy advisers telephoned Reagan at his California ranch Thursday afternoon, the president toughened the language in a statement they had written so it demanded in more forceful terms an end to apartheid, officials said.

But, they added, Reagan did not express disappointment at the turn of events. The president, who did not watch Botha's televised speech at his mountaintop ranch near here, said he wanted to wait and see whether negotiations took place between Botha and his opposition.

One official described Reagan's reaction as characteristic of his tendency to find a ray of optimism amid what seems to others to be a major setback. It was consistent with his long-held view that, although apartheid is repugnant and should be changed, the United States should not try to push South Africa's white leaders into making changes that could destabilize their society.

Even as they described Reagan's views, some informed White House officials expressed their own dissatisfaction with the week's developments in South Africa. One official called them "very disappointing" and a "missed opportunity."

On Thursday, national security affairs adviser Robert C. McFarlane said the reaction of blacks in South Africa would be the most important measure of Botha's speech.

Later, noting the sharp criticism of Botha's address from Anglican Bishop Desmond Tutu and others, a senior White House official said it appeared that the speech had failed to bring negotiations any closer.

Some officials also said privately that the South African government mishandled events by first raising expectations of dramatic change, then dashing them.

The expectations were created in part in a Vienna meeting earlier between McFarlane and Foreign Minister Roelof F. (Pik) Botha.

In that session, the foreign minister made a "persuasive, credible presentation" that reforms were "necessary and right," according to a participant.

A week ago, this participant said the South Africans in Vienna "are very conscious that they must make significant change to respond to black wishes for expression and influence on those who govern them, on citizenship" and other matters.

Similar expectations for sweeping changes were privately relayed to journalists by South African officials.

But the expectations were not fulfilled in Botha's defiant, combative address Thursday. The South African president said he would not lead South Africa's white minority "on a road to abdication and suicide." Botha did refer to citizenship for blacks but rejected other ideas for power-sharing with the black majority, and denied that that majority even existed.

A senior official, asked why the address had failed to live up to the possible reforms that had been indicated, said there are two explanations.

A cynical explanation, he said, is that Botha shied away from sweeping concessions because he believed that "the ballyhoo of it would have brought such a strong backlash" from whites that he would have been forced to retrench.

A more optimistic explanation, the official said, is that South Africa is committed to negotiated changes but wanted to hold off on announcing concessions now so they could be used as "leverage" in actual negotiations later. He said the South Africans may have wanted to let black leaders announce the changes and take credit for them.

However, the official indicated that he gave more weight to the first explanation than to the second.

The sense of disappointment in the administration also reflected a feeling of exasperation toward South Africa. "They are so different," one informed official said of the white minority. "They won't bend an inch if they think their basic values are threatened."

Yet another official said the seemingly inexpert diplomacy exhibited by Botha was characteristic of the South Africans.

White House officials said they learned the night before Botha's address that it was not going to meet the expectations for sweeping reform. They said McFarlane then went to work with officials in Washington to draft a response.

Next day, McFarlane listened to Botha's address and then telephoned Reagan at his ranch in a conference call including Secretary of State George P. Shultz and Vice President Bush. They told Reagan what Botha had said.

It was in this call that Reagan suggested the statement be toughened to say apartheid should be ended, but also said the United States should avoid direct criticism of Botha's speech.

Botha's combative speech is expected to intensify pressure in Congress for legislation that would impose economic sanctions on South Africa. The only step remaining is Senate passage of a conference committee version after Labor Day. While Reagan has not made a decision, some White House officials think he will veto it, but congressional supporters are confident that they can override a veto.