President Reagan told Congress yesterday that there is no sign of "significant progress" toward ending apartheid in South Africa but refused to recommend the imposition of additional economic sanctions.

His refusal to take further action seems likely to provoke another acrimonious battle between the White House and Congress over U.S. policy toward South Africa and to lead to congressional action to impose more sanctions nonetheless.

Indeed, Rep. Howard E. Wolpe (D-Mich.), chairman of the House Foreign Affairs subcommittee on Africa, yesterday announced that "a new legislative initiative" will be launched next week to impose additional economic sanctions on South Africa. Hearings are scheduled to start in the House Oct. 20 and in the Senate two days later.

"The sanctions would have achieved more if this administration were backing sanctions at the U.N. and in other ways," Sen. Paul Simon (D-Ill.), chairman of the Senate subcommittee on African affairs, said. "The White House has been dragging its feet rather than leading against injustice."

Reporting to Congress a year after the enactment of the Comprehensive Anti-Apartheid Act, Reagan said South Africa is "not any closer in late 1987 to respecting free speech and free political participation by all its citizens than it was one year ago."

"I regret that I am unable to report significant progress leading to the end of apartheid and the establishment of a nonracial democracy in South Africa in the 12-month period since the enactment of the act," he said.

But the president, who tried to block the act, said he had concluded that imposing more economic sanctions on South Africa now "would not be helpful in the achievement of the objectives which Congress, the American people and I share."

The impact of the sanctions mandated by the antiapartheid act has been "more negative than positive," the president said, adding that they have increased unemployment for black South Africans and worsened that country's economic stagnation.

The withdrawal of U.S. companies, the Reagan report estimated, cut direct U.S. investment nearly in half, from $2.4 billion in 1982 to about $1.3 billion last year and "probably less than $1 billion" now.

"While our sanctions have accentuated the overall economic stagnation in South Africa, . . . their impact on the government itself and its political choices have not advanced our goals," Reagan said.

Reagan proposed instead that the United States begin "a period of active and creative diplomacy" to bring about negotiations between the white South African government and nonwhite groups on a new political system.

The antiapartheid act, passed by Congress over a presidential veto last Oct. 2, banned the import of South African textiles, uranium, iron and coal; ended new U.S. loans and investment in South Africa, and prohibited South African Airways from landing in the United States.

It also set forth yardsticks by which the president was to measure whether the South African government makes any significant progress toward ending apartheid.

These included repeal of the state of emergency; reforms leading to a nonracial democracy; the release of imprisoned black leaders Nelson Mandela and Walter Sisulu; the start of negotiations with nonwhite groups on a new political system, and an end to raids into neighboring black countries.

If the president concluded none of these steps had been taken, the legislation required him to recommend which of the following additional sanctions be enacted: a ban on the import of South African steel, diamonds or strategic materials; a halt of U.S. military aid to countries such as Israel supplying arms to South Africa, or a ban on South Africans holding bank accounts here.

A ban on steel imports has already been imposed.

The Reagan report, in a review of developments over the past year, generally concluded the situation had grown steadily worse and that violence between the white minority government and its opponents "has, if anything, gotten worse."

The only positive development it noted was the continuing ferment within the white South African community over the issue of power sharing and negotiating with the majority black population. But even there it said no "realistic formula" had been found as a basis for serious negotiations.

The report acknowledged that U.S. sanctions had had some impact on the South African economy and sent "a clear message" of American public outrage against apartheid. It also said they had helped broaden U.S. contacts with blacks.

But it argued that additional sanctions would be "futile" because South Africans are adept at evading such measures, and that new sanctions would be "harmful" to U.S. strategic and economic interests.

"Our sanctions measures have made it more difficult for the United States to persuade the South African government to act responsibly on human rights issues, to move toward negotiations and to restrain its behavior in the region," the report said.

In the "heat of debate" over sanctions last year, it said, "Americans on both sides of the issue overestimated the importance of the United States as a factor in the South African matrix."

What is needed, the report concluded, is "a period of active and creative diplomacy bilaterally as well as in consultation with our allies and friends in Africa" to encourage "meaningful negotiations" inside South Africa for the creation of a nonracial democracy.