The extent of opposition in the House of Representatives to the Reagan administration's request for $8.1 billion in increased resources for the International Monetary Fund has sent a shock through the key lending agency.

Although the administration, strongly supported by the Federal Reserve System, hopes it will eventually succeed in getting the legislation through the House--it already has passed the Senate--a worried IMF for the first time is privately articulating the dreaded question: What happens if the legislation fails?

The danger, says an expert close to events, "is that the IMF would lose its clout as a policy-making institution.

"You can see how it would happen: Country A, that otherwise might come to the IMF to borrow, accepting tough conditions, will figure that the IMF doesn't have enough money. At the same time, the fund won't be able to go to the commercial banks and say, 'We'll go into Country A with such-and-such an amount, how much will you put in?'

"In a word, the IMF will become a toothless wonder."

Of the $8.1 billion, $5.8 billion represents a 48 percent increase in the U.S. quota--deposits--in the IMF, and the balance is a larger contribution to the General Agreements to Borrow, a hard-currency emergency standby fund that the IMF can also draw upon.

Those close to the IMF scene say there are two main "scenarios" that could flow from an American failure to take up its additional $5.8 billion quota.

In the first scenario, the other nations would proceed with their own boosts in quotas. Originally, as agreed by the governing board last February, a 48 percent increase was to have raised the IMF's total capital from about $64 billion to nearly $96 billion.

The increase would have provided the bulk of additional resources for the IMF for several years, beginning in 1984, although the possibility of borrowing from rich nations or private money markets was (and is) always in the picture.

If all the other member nations carry out their commitments for larger quotas, the IMF's total capital would become about $90 billion, absent the U.S. $5.8 billion.

In this circumstance, not only would the IMF be short $5.8 billion in lendable U.S. dollars, but also the existing U.S. quota of about $13 billion would slip just below the 15 percent required for a veto on key issues. Currently, the United States has roughly 19 percent of the quotas, well over the margin needed to have the main say on major policy issues.

In a second scenario, the other major nations may decide to wait and see what ultimately happens in Congress. At best, this would delay building IMF resources at a time that Managing Director Jacques de Larosiere says its "liquidity position is under considerable strain." At worst, the IMF could face a permanent shortfall of contributions.

In either case, the IMF would be left with less resources than it had anticipated, and borrowing outside of quotas would become more urgent. But whether it would have success in borrowing money, given its less secure status, would be a question mark, international experts say.

Former assistant Treasury secretary C. Fred Bergsten said in a telephone interview yesterday that "the IMF role is so crucial that to jeopardize the IMF's financing could unravel the whole international financial system."

He cites as an example the bitter division within the Brazilian government over accepting the tough conditions the IMF set down for its loan to that heavily indebted country. "If the United States now backs away from supporting the fund," says Bergsten, "those Brazilians who wanted in the first place to say 'to hell with the IMF' will be enormously strengthened."

In that circumstance, he said, Brazil might declare a moratorium on all of its debt. "All you need is one big one to touch off others," he warned.

He added that the United States not only risks losing veto power in the IMF, but also could be yielding its claim to leadership in world financial affairs. "It can take a decade to repair the damage," Bergsten said.