Treasury Secretary Donald T. Regan yesterday called for an overhaul of the international monetary system because the global problems caused by recession "continue to grow more serious."

Regan said he will ask finance ministers of the major industrial nations to consider ways to ensure a faster and more coordinated response by governments and lending agencies to cope with critical credit shortages in the Third World.

His unexpected announcement appeared to be one more reflection of an increasingly grim assessment by the Reagan administration of the world economic outlook since the annual meetings of the World Bank and the International Monetary Fund in Toronto in September.

Regan said he would bring up his proposal for the first time Thursday at a dinner meeting in Frankfurt, West Germany, with his fellow finance ministers from West Germany, Britain, France and Japan.

To achieve the end result he favors, Regan said a worldwide conference might have to be called to set it on track. But he stressed that "we are not yet ready for another Bretton Woods" -- site of the 1944 conference setting up the IMF and World Bank. "We are still groping toward a solution," Regan said.

In answers to questions, Regan said that one purpose of a reorganized system would be to introduce more stability in the exchange rate system, although he denied he was advocating a move from present fluctuating rates back to the old pattern of fixed rates.

Since the World Bank and IMF meetings in September, unemployment has increased almost everywhere, and pressures have accelerated for emergency measures to prevent defaults in the Third World that might also threaten the commercial banking system in the richer nations.

The Reagan administration has already approved, in principle, a major increase in funding for the IMF, whereas in Toronto, its insistence on merely a minor increase created a stalemate.

But yesterday, Regan told reporters that the present "ad hoc" basis of relations among the IMF, the Bank for International Settlements in Basel, and the major nations' central banks that help finance Third World problems must be abandoned so there can be "a faster response" to the needs of less fortunate nations.

The "Group of Five" session of the leading finance ministers and central bank heads to take place in Frankfurt Thursday is one of a regular series. This one is expected to make a final decision on the extent to which IMF quotas -- deposits paid into the IMF by members -- should be boosted as one way of enlarging the IMF's lending potential for the rest of the 1980s.

Third World nations have accumulated an estimated $500 billion to $600 billion in debt to the industrial nations and their commercial banks. Without additional loans from the IMF and the commercial banks, some defaults are possible: already, much of this debt has had to be "rescheduled," a euphemism for postponement and softer terms.

Regan made his announcement at a media briefing on his trip planned for this week to Frankfurt, and to Brussels for trade discussions with the European Community. His ideas are not committed to paper, he said, and will be presented to the five finance ministers "on an informal basis."

Regan said that his initiative was a personal one, not yet adopted by the Reagan administration. But he said he had discussed it informally with his Cabinet colleagues.

Pressed for details, Regan responded repeatedly that "I'm just trying to get the thinking started on this issue . . . . The groundwork has to be laid for some kind of financial meeting, it will take a lot of time."

Regan also officially confirmed that the United States seeks a meeting of the IMF's policy board, the Interim Committee, in late January or early February, 1983, instead of the planned April date, to settle on the quota issue.

As a companion measure, it is expected that the Group of Ten richest industrial nations will also increase the funds they make available through the IMF. The quota and the Group of Ten moves together would boost IMF hard-currency resources by nearly $30 billion.

But Regan's proposal yesterday for a revamping and reorganization of the international monetary system -- although still preliminary -- went far beyond the immediate problem of boosting IMF resources.

He appeared to have been influenced by his trip to South America last week with President Reagan, where he said they observed that the impact of the recession, which stimulates the forces of protectionism, "is even more prounounced" than it is here or in Europe.

"The less developed countries who have to go to the IMF for help are all told one thing by the IMF, which is that conditions require that they export more and import less.

"But that creates a puzzle, because the industrialized nations are trying to export more and import less. How can every nation do this simultaneously and still maintain an international trading system?" he asked rhetorically.

He responded that "the only thing that can be done is for all of us to share the misery, and not try to insulate ourselves from it."

Regan said that the pressures brought about "in this period of disinflation" result in a series of uncordinated moves -- consideration of the IMF quota increases, and "bridging" [temporary] loans by the BIS. "There's a lot of ad hoc to this, like our own loans to Brazil and Mexico," he said.

"There's no overall symmetry -- I'm not knocking the IMF, but we don't get the [needy] nations to the IMF soon enough."