South African Foreign Minister Roelof (Pik) Botha has informed the Reagan administration that a United Nations military force is no longer acceptable as a means of policing a transition to independence in Namibia, diplomatic sources said yesterday.

The South African position, which became known as Botha met with President Reagan and members of Congress, is likely to create major new complications for the already embattled international effort to negotiate independence under free elections for mineral-rich Namibia (Southwest Africa).

A U.N. military force of 7,500 men known as UNTAG (United Nations Transitional Assistance Group), previously agreed to by all sides, was an integral part of the plan drawn up by five western nations, including the United States, and approved by the United Nations. It is unclear whether the five-nation group, which is scheduled to meet here last next week, will agree to continue its efforts in view of this latest shift.

The new South African position, as conveyed to Reagan administration officials in two days of talks with Botha, is that U.N. "bias" in last month's Security Council debate disqualified the United Nations from policing a transition.

The Security Council refused to hear the leader of the Democratic Turnhalle Alliance, a local group with close ties to South Africa, while granting the floor to the Southwest Africa People's Organization, SWAPO, a black group waging a guerilla war for control of the territory.

Botha told the Americans, according to the sources, that the decision to reject U.N. forces was that of the Democratic Turnhalle Alliance, and is supported by South Africa.

The United Nations still might be able to have a civilian presence to assist in a territorial, Botha is reported to have said. The South African theory is that an election can work only if Angola and the SWAPO forces residing there cooperate to the extent of moving the SWAPO guerrillas north, away from the election area in Namibia. In this case, civilian monitors might be adequate.

The nature of the transitional police presence is of major political as well as military importance. In the South African view, a U.N. force seen by the populace to be favoring SWAPO could tip the scales for that group in elections.

Neither U.S. nor South Africa officials would provide details of Botha's White House meeting with Reagan, arranged at the president's request. Acting White House press secretary Larry Speakes called it "a friendly meeting" that covered "the general policies of southern Africa."

Botha met with Secretary of State Alexander M. Haig Jr. Thursday and continued detailed discussions about Namibia yesterday with Chester A. Crocker, who has been nominated to be assistant secretary of state for African affairs. Following his meeting with Haig, Botha said he sees "a very real possibility" of making progress on a negotiated Namibian settlement within the framework outlined by the U.S. officials.

As interpreted by South African sources, the U.S.-sponsored framework includes:

A Namibia that is nonaligned between major power blocs.

Written guarantees, whether by constitution or some other declaration, of fundamental rights of citizenship as well as minority rights in an independent Namibia. In the South African view, these should include the right to regular elections.

A U.S. pledge not to pressure South Africa to accept solutions it believes are counter to its interests.

A declaration by Haig that no deadline will be set in connection with the Namibian negotiations.

In a meeting with members of the Senate yesterday morning, Botha is reported to have said the the process of working out the constitutional guarantees with Namibian parties could take up to two years. He is said to have made no statement of how long it would take to achieve a vote and full Namibian independence.