Japan's moves over the past three years to open its booming markets to foreign products have failed to live up to their highly touted promises and produced few sales for American businesses, according to a Reagan administration study.

Tariff cuts were described as "too small" and "quite shallow" and as having "marginal" and "limited" impact on efforts to sell more American products, while steps to remove nontariff barriers failed to increase the value of foreign manufactured goods sold in Japan, the study found.

A $1 billion increase in the value of foreign goods sold in Japan between 1980 and 1983, for example, "largely reflected price increases," the report said.

The study was prepared for the Senate Foreign Relations Committee in October by the State Department and the Office of the U.S. Trade Representative. Although it is not classified, neither administration officials nor committee members have made any effort to disseminate it and the report is available only in photocopy form.

Secretary of State George P. Shultz concluded in a letter sending the study to the Senate committee that five Japanese initiatives over the past three years have "improved" the access for foreign goods in Japan, but "significant barriers to trade still remain."

The study takes on increased significance as the Reagan administration prepares for the Los Angeles meeting next Wednesday between President Reagan and Prime Minister Yasuhiro Nakasone.

The Japanese are considered unlikely to offer new trade concessions during that meeting, but, instead, are likely to emphasize the past market-opening initiatives covered in the report.

Japan's ballooning trade surplus, expected to reach a record $35 billion this year and to go even higher in 1985, is a growing source of friction between the United States and its closest Asian ally.

Some high administration officials are concerned that the trade frictions will undermine a growing closeness between the world's two strongest economies and overshadow attempts to run a common course in the Pacific basin.

Nonetheless, it remained unclear whether Reagan will follow the advice of some of his principal economic advisers and get tough about Japan's trade practices during his meeting with Nakasone, with whom aides say he has developed a close "Ron and Yasu" relationship during their four previous meetings.

"How can I get tough with a very good friend?" Reagan asked after a National Security Council meeting Friday to lay out options on the trade issue.

Reagan administration trade officials have grown increasingly frustrated over the vast gulf that exists between Japanese promises and the reality of trying to sell more American-made products in Japan. This gap comes across clearly in the 50-page report.

Japan, for instance, took nearly two years to keep a promise made with great fanfare in January 1982 to end restrictions on imports of American-made softball bats -- a market segment that U.S. manufacturers believe they have a good chance at cracking.

Similarly, Japan took 22 months to keep its January 1983 pledge to make it easier for American companies to sell farm chemicals by setting lab standards for testing.

Tariff cuts, furthermore, were described as selective, and duties "on individual items remain high, particularly on certain manufactured and high value-added agricultural products. . . . Many items remain on which tariff cuts would afford a significant increase in market opportunities for U.S. exports," the report said.

And despite a commitment made last April, Japan's Ministry of Health refuses to accept clinical tests done on non-Japanese patients for medical devices, delaying sales of sophisticated American-made diagnostic equipment.

Although new laws were passed to ease Japanese standards and certification practices, "full implementation at the administrative level has not been achieved.

"Because progress has been slow, many of the policy pronouncements included in recent announcements repeat previous commitments regarding standards and certification," the report said.

It added that Japanese commitments "have not been specific or remain unimplemented" in the areas of government procurement, import promotion and participation in its market of service industries such as banking, law firms, shippers and stock brokers.

The report said that commitments to open the Japanese market to increased sales of American cigarettes, telecommunication equipment and financial services "will depend very heavily on the manner in which they are implemented."

U.S. trade officials have noted that new trade barriers appear to be cropping up in high-technology areas where American companies maintain an advantage, and have expressed fears that Japan intends to restrict foreign competition in those markets to give its domestic industries a chance to catch up.

American lobbying earlier this year blocked a potentially restrictive practice on computer software imports, but received no assurances the effort would not be renewed in 1985.