The agreement between the United States and Japan to liberalize Japanese capital markets has failed in its goal of boosting the value of the yen against the dollar, according to a new study by the Institute for International Economics.

Increasing the value of the yen in relationship to the dollar was the chief objective of the agreement reached between the two governments last May.

The author of the report, Jeffrey A. Frankel of the University of California at Berkeley, said that "no amount of fiddling with the Japanese capital market" can solve the problem of the weakness of the yen, which he said is caused by an outflow of capital attracted by high U.S. interest rates.

Frankel, a former aide to Martin S. Feldstein when he was chairman of the Council of Economic Advisers, said the yen has weakened 6.5 percent against the dollar since the agreement was signed, even though the Japanese government adopted specific capital market liberalizations pressed on them by the United States.

At a press conference yesterday, Frankel said, "the Treasury probably got caught up in the excitement, and didn't pay attention to the economic logic" of their proposals. Secretary Donald T. Regan, Frankel said in his report, "often appeared to miss altogether the point that allowing more capital outflow from Japan would weaken the yen."

Since the agreement, the projections of the two nations' trade imbalances have grown larger. Institute Director C. Fred Bergsten forecast yesterday that the U.S. global deficit, now running at a record $130 billion rate, will touch $150 billion in 1985. The U.S. trade deficit just with Japan was calculated at $35 billion this year and $40 billion for 1985.

"These numbers are becoming serious," Bergsten said, "and I think Japan is running a risk to let the deficits grow and fester." He predicted that unless President Reagan and Prime Minister Nakasone address the problem at their California summit on Jan. 2, the United States may drift toward protectionist solutions, such as a new tariff surcharge on imports.

The thrust of Frankel's study is that while opening up Japanese capital markets is a healthy thing from the standpoint of the Japanese economy, the yen/dollar agreement has failed in the goal set out by the United States, which was to defuse the trade deficit problem by driving up the value of the yen.

"Capital is flowing out of Japan and into the United States, weakening the yen, strengthening the dollar, and creating record current account imbalances, primarily because real interest rates are higher in the United States than in Japan," Frankel said.

"This is in turn because national saving is much lower in the United States than in Japan, a divergence that the recent trends in the two countries' fiscal policies have accentuated."

Frankel goes on to conclude that the "yen is not especially undervalued" despite Reagan administration assertions to the contrary. Some American officials and many businessmen perceive the Japanese yen as "too cheap," exacerbating the difficult competitive position of American exporters. The Japanese, on the other hand, tend to agree with Frankel's thesis that the problem resides in a dollar too strong because of U.S. economic policies.

Institute staff member Stephen Marris added that the Japanese contribute to the "structural" problem by generating a huge surplus of savings -- the opposite problem to the one in the United States. Japan could absorb some of that excess savings amount by a more expansionary fiscal policy, he said.

These differences over macro-economic and trade policies will be high on the agenda of the Reagan-Nakasone summit. Other issues, including auto quotas, steel imports into the United States, and American pressure on Japan to buy more foreign manufactured goods, are also expected to come up.

The yen/dollar agreement, negotiated by the Treasury and Ministry of Finance officials as an outgrowth of a November 1983 Reagan-Nakasone meeting, removed remaining barriers to outflow of capital from Japan; encouraged "internationalization" of the yen; and secured more favorable treatment of American banks and other foreign institutions doing business in Japan.

When the agreement was announced, Treasury officials acknowledged that the yen would not necessarily appreciate in the short run, but argued that, over the long haul, internationalization would cause the yen to move up.

"Exchange rates are determined by many unpredictable factors," Frankel observed. "It may be that no one will ever be able to tell whether the value of the yen is higher on account of the 1984 liberalization than it otherwise would have been."