The South African government, in a move widely believed to reflect a new responsiveness to American diplomacy since the Reagan administration came to power, has backed off from plans to license journalists and thus increase controls over the press.

Instead, the newspapers are to set up a new media council of their own design, with powers to reprimand and fine papers that breach a code of conduct but not to strike journalists from a register of practitioners.

The government is to recognize the new council formally. Editors have expressed concern that this may open the way for an indirect system of governmental control of the press.

For the moment, however, most editors agree that it is not only far less ominous than the threatened licensing system but actually may improve their situation.

David Dalling, spokesman on media affairs for the opposition Progressive Federal Party, thinks the Reagan administration's influence was a major factor that led the government to hesitate to implement its original plan for restricting the media.

"This is difficult to quantify," Dalling said, "but I know the Americans have made a prominent issue of press freedom, and the government is reluctant to do anything that will cool its relationship with the Reagan administration."

In one sign of U.S. interest in the issue, U.S. Ambassador Herman Nickel visited Parliament to listen to debate during a dramatic, all-night session on press controls.

Dalling says the policy of "constructive engagement" with South Africa followed by the Reagan administration, in contrast with the more publicly critical line taken by the Carter administration, has given the United States greater diplomatic leverage.

"The warmer relationship is very important to the government," he said. "It means it responds when the Americans make noises, and they have been making noises over the threat to the press."

A number of journalists agreed. Rex Gibson, editor of the Rand Daily Mail, most outspoken of the English-language dailies, said, "We don't have concrete evidence of American pressure, but everyone assumes this was a major factor in causing the government to back off."

The other major factor cited by Dalling for the government's action was opposition to the proposed press curbs by progovernment Afrikaans-language newspapers as well as English-language newspapers, which generally support the opposition.

Peter McLean, chairman of the Newspaper Press Union, the publishers' organization, said the solidarity shown by Afrikaner publishers with their English-speaking counterparts was decisive.

The new media council, designed by journalists, will replace an earlier one, established under government duress, which many journalists regard as a virtual court.

The new system is the result of five months of bargaining between the press and the government since an official commission recommended the licensing system on Feb. 1.

The commission, headed by a member of a provincial supreme court, Judge Marthinus Steyn, presented a draft law making it a crime to employ an unlicensed journalist or to publish a report from any journalist not enrolled by a statutory body called the General Council of Journalists.

This general council initially would be appointed by the government and would have the power to fine journalists or strike them off the roll.

The proposal was the culmination of years of government threats to bring South Africa's stubbornly independent press under tighter control.

There are more than 100 statutes limiting what newspapers may publish, but skillful editing and legal advice have enabled South Africa to retain what the International Press Institute described in 1980 as the freest press in Africa.

In the outcry that followed publication of the draft law, the government hesitated to implement it. The union then began a series of informal meetings with the man responsible for implementing it, Interior Minister Chris Heunis.

"Heunis kept probing to get a chink to open up in our uniform opposition, but none appeared," said McLean.

Eventually Heunis agreed to drop the draft law if the press would improve its system of "self-discipline."

The union told Heunis it would replace the old press council with this new one, and he agreed. In a last-minute hitch, Heunis suddenly introduced another press law June 11 to make the new media council a statutory body and force all newspapers to submit to it by joining the Newspaper Press Union.

This was a partial resurrection of the earlier plan for a statutory council. It was also aimed at controlling newspapers run by right-wing political parties that have broken away from the ruling National Party and whose growing influence among Afrikaners the government fears. These party papers do not belong to the press union.

Facing strong opposition, Heunis later withdrew the clause making the media council a statutory body. The law still states that the minister must recognize the council, however, and this is what worries some journalists.