Two House subcommittees yesterday voted to limit sharply new U.S. business investment in South Africa because of that country's racial segregation policies.

By voice votes, they approved legislation that would prevent U.S. companies from investing new capital there and would allow investments only of profits earned there by existing U.S. firms.

The legislation had been opposed by the Reagan administration, which has launched a major effort to increase trade and improve relations with South Africa. If enacted, experts said, it could have a major effect in curbing American investments and operations in South Africa.

The bill, authored by Rep. William H. Gray III (D-Pa.), was approved by the House subcommittees on African affairs and on international economic policy and trade. Gray acknowledged later that the bill faces an "uphill fight" in the Foreign Affairs Committee.

He argued before the subcommittees that investments by about 500 U.S. corporations help support South Africa's system of apartheid and are offensive to African blacks and citizens of Third World countries.

The legislation would provide for jail sentences and fines for companies that violate the restrictions. It would permit the president to lift the prohibition on new investment by certifying that South Africa is making progress in promoting human rights.

The two subcommittees also approved a bill by Rep. Stephen J. Solarz (D-N.Y.) requiring all American corporations operating in South Africa to adopt mandatory fair-employment practices for black workers similar to U.S. civil rights laws.

It would require equal pay for U.S.-paid blacks and whites, no discrimination in employment and elimination of inequities in seniority between the races.

A system of voluntary fair-employment practices is currently supposed to be observed by American companies operating in South Africa. Critics have contended that it is observed by only about half.