The major nations of the International Monetary Fund, including the United States, broke a deadlock early today over an American proposal to limit the borrowing "access" of member nations to the IMF money pool next year and for several years thereafter.

In a session after midnight, the finance ministers agreed to a compromise that will limit borrowings by member nations to 102 percent of their quotas (deposits) in the IMF over the next three years. This in effect will limit borrowing to no more than is available under the current rules, and was the principal demand set forth by Treasury Secretary Donald T. Regan.

But Regan yielded to the demands of the European, Japanese and Canadian members who insisted that a more generous allotment be given to countries with severe balance-of-payments problems. Therefore the compromise will allow member nations in such circumstances to draw 125 percent of their annual quotas for a three-year term, or a maximum of 375 percent.

The IMF Interim Committee, announcing the agreement after daylong meetings, observed that a few nations had some reservations about the compromise. Principal objections apparently were raised by China, India and some African nations.

IMF officials said it was important to have reached a compromise, because a continued deadlock would have shaken worldwide confidence in the agency's ability to manage the debt crisis and would have exacerbated the trend of commercial banks' reducing their lending in Third World countries.

The compromise was hammered out at a long dinner meeting after Regan had precipitated the deadlock by refusing to budge from his proposal to limit borrowing next year to 102 percent for any nation.

In a tough speech at the opening of yesterday's session, Regan conceded that the U.S. proposals would result in an absolute reduction of borrowing potential for 108 of the IMF's 146 members. "But," he said, "it's time we got the facts straight . . . . The 1984 reduction in maximum access for the great majority of these 108 countries is very small, averaging only 5 to 8 percent."

Originally, an IMF member was allowed to borrow hard currencies--dollars or yen, for example--equal to 100 percent of its quota, or deposits of its own currency in the IMF.

As the debts of poorer nations mounted, borrowing access was increased to 125 percent of quota, and finally, in 1980, to 150 percent a year for three years. The quotas are being raised by an average of 48 percent, meaning that each member will increase its contribution.

Regan had proposed that for 1984, maximum borrowings be limited to 102 percent instead of 150 percent after the quotas are increased by 48 percent.

The compromise worked out last night was similar to a British suggestion made at a luncheon meeting. But at that point, Regan argued that he had to come away from yesterday's session with a specific understanding that borrowings beginning in 1985 would be gradually phased down. It is this element that the Americans seem to have backed away from in the final compromise.

After arguing all morning and through lunch about the potential impact of various access levels, the Interim Committee asked IMF Managing Director Jacques de Larosiere to produce estimates showing how various figures below 150 percent would affect borrowers.

The 22 finance ministers that make up the Interim Committee then scheduled the unusual night meeting to try to break the deadlock.

Meanwhile, the United States yesterday refused to contribute more than $750 million a year for IDA-7, the International Development Agency's lending program that would start July 1, 1984.