The House Foreign Affairs Committee set in motion a new congressional confrontation with the Reagan administration over policy toward South Africa yesterday by approving legislation that would ban new U.S. loans and investments there.

The committee approved the measure despite strong objections by Secretary of State George P. Shultz, who warned that it could undermine administration policy and increase the chances of violent clashes in the racially torn country.

"We do not believe it should be our purpose to harm the South African economy, nor do we believe that such action will hasten the end of apartheid," Shultz said in a letter to committee Chairman Dante B. Fascell (D-Fla.).

"In fact, I am concerned that such a measure could have the opposite effect, heightening intransigence on both sides at a time when we should be strengthening voices of dialogue and moderation," he wrote.

But committee Democrats declared the administration's policy of "constructive engagement" toward South Africa a failure and said that, without the threat of increasingly tougher U.S. economic sanctions, the white-minority government would continue to refuse to negotiate with the black majority.

"Very simply, if the South African government refuses to bring this problem to the [negotiating] table, we are going to find an explosion in South Africa," said Rep. Howard E. Wolpe (D-Mich.), the chairman of the Africa subcommittee that drafted the bill.

The committee's 25-to-13 approval of the bill set up the possibility of a repeat of last year's clash over South Africa. In September, President Reagan reluctantly imposed mild economic sanctions after legislation containing more stringent measures had cleared the House and was on the verge of Senate passage.

The new bill is expected to be on the House floor next week and is thought likely to be passed. Similar Senate legislation, sponsored by Lowell P. Weicker Jr. (R-Conn.) and Edward M. Kennedy (D-Mass.), has been introduced.

Senate Foreign Relations Committee Chairman Richard G. Lugar (R-Ind.) plans to hold hearings on South Africa in July, according to his spokesman, Mark C. Helmke.

House Republican moderates Jim Leach (Iowa), Olympia J. Snowe (Maine) and Christopher H. Smith (N.J.) voted with a solid bloc of committee Democrats for the sanctions legislation. Leach said that he is not sure the bill "will put us on the right side of history" but that "opposing a tightening of the noose could put the Republican Party on the wrong side of its heritage."

Pressure from GOP lawmakers in both chambers was instrumental in forcing Reagan to impose economic sanctions last year.

The House measure would immediately prohibit new U.S. investments in South Africa except reinvestments of earnings by firms operating there and would ban new loans and other extensions of credit to the South African government or entities it controls.

The bill would also prohibit U.S. participation in energy development there, ban import of South African uranium ore, coal and steel, and deny U.S. landing rights to South Africa's national airline.

In addition, the measure would require complete disinvestment by the U.S. computer industry within 18 months unless the apartheid system of strict racial segregation is dismantled or negotiations begin on that and unless all South African political prisoners are released. These would have to include Nelson Mandela, head of the outlawed African National Congress.

If one condition was not met by June 30, 1988, midpoint in a presidential election year, Reagan would be required to make recommendations on total U.S. disinvestment.

Most committee Republicans opposed the bill, charging that it was being "ramrodded" through the House in a partisan attempt to embarrass the administration.