South African Foreign Minister R. F. (Pik) Botha made a secret trip to Salisbury today for talks with Prime Minister Ian Smith over the new Rhodesian initiative announced earlier this week.

There has as yet been no acknowledgement of the trip by Rhodesian officials - who vigorously denied the first reports - but it is widely believed here that Botha may have made the trip at least partially on behalf of the British and U.S. governments, which are seeking terms to settle troubled Rhodesia's 12-year-old constitutional crisis and five-year war.

In a nationwide address Monday, the Rhodesian leader indicated that he was rejecting the Anglo-American plan - which calls for one-man, one vote elections - as too radical. He said its terms for implementing majority rule deviated from those originally discussed several months ago with British Foreign Secretary David Owen and that he was now looking at alternative means of settling the issue presumably by negotiating to reach a settlement with moderate internal black factions.

Moderate black leaders, however, have rejected Smith's initiative.

It is understood here that Smith's speech took both London and Washington by surprise, and that they, with South Africa, have been left in the dark about the effects of his announcement on Western settlement efforts.

Botha may have come to determine the status of the peace talks and also to make it clear to Smith that the South Africans do not want Rhodesia to reject the Anglo-American effort.

[After his return to Pretoria, Botha told reporters that Smith said he did not intend to break off the settlement negotiations with Britain, Reuter reported.]

South Africa has significant leverage over Rhodesia, since it is the landlocked country's only friendly link with the outside world and main trading partner. South African pressure has already played a significant role in getting the white-minority government at Rhodesia to discuss a transfer to majority rule.

There have been growing indications here that Smith's government was not prepared to accept the Anglo-American proposals because of two basic disputes:

They call for immediate one-man, one-vote. Smith has made it clear that he wants a qualified franchise - on the basis of a combination of property and education requirements. This would in effect leave the 270,000 whites in a strong position.

They call for granting preferential treatment to nationalist guerrillas in the armed forces after the end of the war. Smith is adamant that the current army personnel must remain in control.

A four-member Anglo-American team, headed by British envoy John Graham and U.S. Ambassador to Zambia Stephen Low, have held two series of talks in Rhodesia. Each time they have run up against strong opposition.

Government officials here have charged that the proposals offer no guarantees for whites and are thus unacceptable. Monday Smith said it was clear to him that the British believe that, "They are dealing with a divided and weak Rhodesia and that they have us on the run."

But divided and weak Rhodesia clearly is. Smith is now facing the most serious challenge since Rhodesia broke from Britain in 1965. The dwindling white population is fiercely divided over the settlement plans, forcing Smith to call a national election for Aug. 31 to test his strength.

Twelve members of Parliament recently broke from the ruling Rhodesia Front Party to form a new right-wing movement opposed to black-majority rule.

Meanwhile, the guerrilla war continues to escalate. Army headquarters announced tonight the deaths of 23 civilians in one incident that reflects the increasing ugliness of the war.

A military communique reported that last Friday, a gang of geurrillas forced an African man, his nine wives and 13 of his 36 children - who ranged in age from two years to their teens - into a hut, locked the door and set fire to it. The icident occurred at Rushing, 15 miles from the Mozambique border. The report said, "Knowledge of the atrocity only reached the security forces early today because there were no survivors."

The Rhodesian armel forces are now stretched thin because of the two new fronts on the Zambian and Botswana borders. That means that of the 1,800-mile border only about 120 miles - the part shared with South Africa - is not vulnerable to guerrilla attack.

The much-quoted "kill ratio" - indicating the number of deaths of troops versus guerrillas - has fallen dramatically in the recent months, from over 10 to one, down to three guerrilla deaths for very Rhodesian soldier.

Guerrilla forces are now crossing the border in bands of from 40 to 60, compared with the small squads of eight to 12 earlier this year. The Rhodesians recently charged that the numbers of guerrillas and the new sophistication of their equipment had forced the army to cross the Mozambique border in hot pursuit operations to hit Nationlist camps before they could attack Rhodesia.