The United States and South Africa are engaged in intensive negotiations which the Carter administration hopes will result in South African agreement - perhaps by the end of the week - to sign the Nuclear Non-Proliferation Treaty.

U.S. Ambassador Gerard C. Smith, the administration's top negotiator of nuclear agreements, arrived in Pretoria Sunday to open secret talks with South African leaders. His presence in South Africa became known yesterday.

South African agreement to sign the NPT - which formally commits nonnuclear weapon states not to build nuclear explosives, and to accept mandatory international safeguards on all their peaceful nuclear facilities - would represent a dramatic triumph for non-proliferation efforts.

The Carter admin stration has been particularly anxious to persuade South Africa to sign the non-proliferation treaty in view of the highly publicized scare last August over indications that South Africa might be preparing to test a nuclear weapon.

South Africa denied any intention of producing or testing a nuclear device, but most experts believe South Africa has the ability to build nuclear nium that South Africa needs for its weapons - if it has not already done so.

Neither U.S. nor South African officials would provide any details yesterday of the negotiations now under way. A. J. A. Roux, president of South Africa's Atomic Energy Board, termed that talks "too important and too sensitive" to discuss.

But informed sources said SMITH WAS TRYING TO WORK OUT AN ARRANGEMENT UNDER WHICH South Africa would promise to sign the treaty in return for U.S. agreement to provide fuel for South Africa's nuclear research and power reactors.

While some sources suggested that a final agreement might not be concluded this week, they noted that the administration's decision to send Smith - who negotiated the SALT I pact with the Soviet Union - attested to the advanced state of the talks.

Officials also observed that the United States is not without leverage in this affair.

The Carter administration has been stalling ever since it took office on supplying a shipment of 57 pounds of highly enriched weapons-grade ura-Safari I nuclear research reactor at Pelindaba, just west of Pretoria.

South Africa has also been informed that U.S. firms will not be allowed to provide the slightly enriched uranium hexaflouride needed to operate two nuclear power plants now under construction outside Cape Town unless Pretoria signs the treaty.

South African Prime Minister John Vorster has repeatedly stated that his country is only "interested in the peaceful applications of nuclear power," and officials have also stressed that South Africa does not object to the principle of non-proliferation.

Both Pretoria's objections to the treaty - and the concern in the United States and other countries over South Africa's nuclear program - have centered on South Africa's secret uranium enrichment plant at Pelindaba.

The small experimental plant, which was completed in 1976, is presently capable of enriching small amounts or uranium using a new process South African developer.

The facility is currently not subject to international safeguards or inspection by the International Atomic Energy Agency. While it is not known whether weapons-grade uranium can be - or is being - produced at Pelindaba, at present, South Africa earlier this year announced plans to convert the facility into a plant capable of producing enriched uranium in commercial quantitites.

The United States wants to see the Pelindaba enrichment plant placed under "full-scope" safeguards that would allow South Africa to produce enriched uranium for power plants, but prevent production of weapons-grade uranium.

South Africa, on the other hand, expresses concern that if it opens up its enrichment plant to international inspection, the "unique" process it developed may be stolen.

While South Africa has long intended to operate the two nuclear power plants now under construction with fuel produced at its own enrichment facility, the commercial enrichment plant originally envisaged was not expected to go into commercial operation until the mid-1980s.

Since the two French-built power plants will be completed by 1981, South Africa hoped to obtain fuel from the United States for at least the first three years of operation.

South Africa's decision earlier this year to convert the Pelindaba plant into a commercial facility - rather than built a new enrichment plant - was viewed as designed to speed up production of fuel needed for the power reactors.