The House yesterday began heated debate on amendments to the controversial bill authorizing a substantial increase in U.S. funding for the International Monetary Fund, with passage of the legislation still in doubt.

After three hours of debate underscoring dissension within both parties, the House defeated legislation that would have placed restrictions on the interest rates U.S. banks could charge on loans to Third World countries forced to restructure their debt and the length of payments on such loans.

Although debate was limited to amendments to the bill rather than the bill itself, a number of congressmen spoke out on the overall issue, some defending the proposed $8.1 billion in new U.S. aid as an essential addition to the international financial safety net, others condemning it as a bail-out for major banks or unnecessary aid for foreign countries that should be going to American small businesses and home buyers instead.

A vote on the bill and continuation of debate is scheduled for Thursday.

Congressional sources said there still may not be enough votes to pass the entire bill and it may be put off until the administration can gather more support. The bill has already been pushed back because of lack of votes.

However, an administration spokesman said yesterday, "We think the votes will be there when the votes count." Earlier in the day, House Speaker Thomas P. (Tip) O'Neill Jr. quoted the administration as having said it had 100 votes for the IMF bill, but congressional sources doubted the count was that high.

President Reagan has said an increase in funding for the international financial rescue agency is critical to the international and domestic economies.

Some congressmen said passage of the amendment defeated yesterday would have helped them vote for the entire bill. But congressional sources said the final vote could not be determined from yesterday's debate. Some congressmen who voted against the amendments favored the IMF package, and it was impossible for observers listening to the debate to tell which side had the upper hand on the ultimate question of support for the IMF.

The amendment sparking the most debate was designed to prevent U.S. banks from taking advantage of debt-stricken Third World countries by charging higher interest rates and other fees on new loans.

It would have required the U.S. executive director of the IMF to vote against providing any assistance in rescheduling debt for a country with annual external debt service exceeding 85 percent of its annual export earnings, unless the rescheduling converted short-term bank debt at high interest rates into long-term debt at lower rates.

The legislation would also require that the interest rates, and all other fees charged under the restructuring, to be no more than one-half of a percentage point above the London Interbank Offer rate, which is considered the cost of money for international banks, similar to the U.S. prime rate.

Some of the sometimes heated debate centered on whether the banks should be blamed for aggravating the international debt crisis and how far the Congress should go in regulating American banks.

"I am not out to seek blame," said Rep. Charles E. Schumer (D-N.Y.), sponsor of the amendment. "I am out to prevent what happened before from happening again," meaning banks charging Third World countries high interest rates on their restructured debt.

"For those of you interested in taking a pound of flesh from the banks, vote for this amendment," said Rep. John La Falce (D-N.Y.).

Other congressmen said it was unfair to provide low-interest loans to foreign countries when American small business owners and prospective home buyers faced double-digit interest rates.

"I'm not a representative of Argentina or Brazil or Mexico or any of the other countries" involved with the IMF, La Falce said. Schumer's amendment would have adverse effects on the U.S. economy, La Falce said.

The amendment was rejected by a vote of 268 to 157.

Stephen Neal (D-N.C.) said it was ridiculous to expect banks to make loans at such low interest rates. "Could you do this and make a living?" Neal said. "The banks have no intention of becoming foreign aid institutions."