Fritz Leutwiler, the Swiss mediator in the attempt to solve the problem of South Africa's debt repayments, said today that he was "greatly encouraged" by President Pieter W. Botha's pledge to loosen some of the restrictions imposed under that country's apartheid system.

But the debt problem will not be resolved unless American banks take a similarly optimistic stance, Leutwiler said through a spokesman. He scheduled a meeting for London on Feb. 20 at which all major creditor banks will be asked their position on the debt issue and on a plan Leutwiler is drafting to solve the current crisis.

South Africa has asked for a moratorium on debt repayment through 1990. Leutwiler, a former president of the Swiss National Bank and now board chairman of Brown Boveri, a leading Swiss engineering company, was told by American bankers during a pre-Christmas visit to New York that there is no way that they will accept such a delay, according to his spokesman, Erich Heini. U.S. banks are by far the largest creditors of South Africa.

"The optimism Mr. Leutwiler brought back from South Africa a few weeks ago is fully justified," Heini said. "But the ball is now in the court of the American banks. If they don't agree to some form of compromise based on what Mr. Botha has said, we've advanced no further."

Leutwiler has not spelled out the plan he intends to present at the London meeting, but informed sources said he was seeking a compromise under which debt repayments would begin before 1990 but at a slower pace, enabling South Africa to restore its financial position.

About 30 major banks from the United States, Europe, Canada and Japan are represented on the informal committee that will meet with Leutwiler on Feb. 20.