A major new agreement to manage the international monetary system in order to curb erratic fluctuations in the dollar and other currencies will be announced by the seven economic summit nations Tuesday, Treasury Secretary James A. Baker III said today.

"We have made progress on means for improving the functioning of the exchange rate system," Baker told reporters in describing the latest Reagan administration initiative.

The agreement -- to include frequent reviews of economic performance -- represents the most significant attempt in more than 13 years by the world's leading noncommunist nations to achieve international economic coordination.

If the new system succeeds in making exchange-rate relationships more stable, it will make it easier for businesses to make long-range plans. When exchange rates fluctuate widely, as they have in the past six or seven years, business executives cannot be sure whether their products will suddenly face stiff import competition or whether their exports will be cheap or expensive.

A seriously overvalued dollar in the past two years has helped cause an enormous American trade deficit, making U.S. products more expensive abroad. Now, as the dollar has depreciated, the Japanese yen has turned so strong that Japanese manufacturers fear the loss of their export markets.

The overhaul announced by Baker goes far beyond what administration officials had expected at the Tokyo summit.

A senior administration official described the new system as "a managed float" that will force the large industrialized nations represented here to take remedial action when exchange rates get out of line. The idea is to suggest a midway position between the freely floating system that has existed since 1972, and the more rigid fixed-rate system that it replaced.

Although the plan would involve a degree of government management of exchange rates, the official said, "There is no ceding of governmental sovereignty." The force behind it will depend "on a certain amount of peer pressure," based on endorsement of the plan by the heads of government here, "and a certain amount of public pressure" from businessmen anxious for stable rates.

It was unclear whether the seven industrial nations had agreed today on what they believed were acceptable exchange rates between the U.S. dollar, the world's mostl widely used currency, and the currencies of other major industrial nations.

The official said the new system is so far-reaching that he hoped it would make unnecessary a new international monetary conference. President Reagan earlier this year ordered a study of the need for such a conference, and Baker will continue that task, the official said.

He said the plan helps meet international reform objectives demanded by French President Francois Mitterrand at the Versailles summit in 1982.

The senior official declined to speculate on the impact of the scheduled announcement on the foreign exchange markets.

"This is an impressive step forward," said C. Fred Bergsten, director of the Institute for International Economics in Washington. But he questioned whether the process of changing volatile exchange rates will be automatic enough. "They probably need an independent referee to blow the whistle," he said.

The main pressure for nations to change their policies will develop when and if it becomes evident that a nation's economy is veering away from its own forecasts, as measured against a series of economic indicators, including exchange rates.

Although the new system does not call for publishing each nation's exchange rate targets, the official said he anticipated that "they will become known."

Baker's successful effort to win agreement for a more formalized method of coordinating economic policy was accomplished in the face of initial resistance from the West German and British governments. Bonn and London tend to prefer minimum intervention in the current "floating" system, which allows markets to determine exchange rate relationships.

Baker told reporters after the first day's summit session that President Reagan and other heads of government had approved a "mechanism" that builds on an agreement reached last fall at New York's Plaza Hotel by the Group of Five nations to intervene to bring down the international value of the dollar.

The Group of Five includes the finance ministers and central bankers of the United States, Japan, West Germany, France and Britain. Under the agreement reached today, the group would be broadened to include Canada and Italy when policy discussions affect them.

Canadian Finance Minister Michael Wilson hailed that provision, saying: "Where the interests of Canada are involved, we will be a part of that meeting."

The Group of Five will continue as the key framework for closer economic coordination, not only on exchange rates but on all facets of international economic policy.

Officials said all seven countries will engage, at least once a year, in "surveillance," a self-examination of their performance against their own economic forecasts. The exercise will be coordinated by the International Monetary Fund. The Group of Five nations will examine their performance more frequently.

A summary of the plan, to be included in the final summit communique, says the measurements will include gross national product, inflation, interest rates, unemployment, budget deficits, current account and trade balances, monetary growth and reserves, and exchange rates.

If one or more of the five major countries believes that another's performance is moving out of line, the administration official said, "That would trigger a meeting and a call for remedial action."

He conceded that the new system will be characterized by some as the equivalent of "target zones" -- a plan under which nations would agree to keep their exchange rates within fixed parameters. But he contended that the Tokyo summit plan does not establish specific targets for exchange rates.

Under the proposed system, when the Group of Five call a meeting and urge corrective action on exchange rates, the method of altering the rates would be intervention, along the lines of last fall's agreement.

In effect, he said, "The Plaza agreement is becoming institutionalized, with more surveillance, more meetings, the use of remedial action and indicators -- a more formal procedure."

The administration official said that the agreement endorses the work of the Group of Five as leading to better coordination of exchange rates, and a lowering of interest rates on a basis "that was both orderly and noninflationary."

In effect, that language rejects the complaint by the Japanese government prior to the summit that the appreciation of the yen by 40 percent against the dollar since the Plaza agreement was "excessive." But Japanese officials, commenting on the proposed new system, said last night that "there will be coordinated intervention when it is useful."

The aims of improved coordination would be to promote noninflationary growth; strengthen market-oriented incentives for employment and investment; open the international trading and investment system; and foster "greater stability and less uncertainty in exchange rates," the communique says.

Last summer, as the U.S. dollar began to decline, Baker sounded out the other Group of Five countries on their willingness to undertake a coordinated intervention to keep the dollar decline going, bringing it more into line with what they considered realistic values.

Later, Baker used the same mechanism to encourage the central banks of West Germany, Japan and the United States to lower their discount rates, which further encouraged exchange rate adjustments.

Baker had been seeking a way to refine the process that would lead to further coordination of economic policy without resorting directly to "target zones," rejected out of hand by West Germany and Japan as too close to the old, fixed-rate system.

They argued that intervention in markets -- buying and selling each other's currencies to defend the zones -- would be too expensive. Baker appears to have something close to target zones with the new system, but without the precise name.