U.S. Treasury Undersecretary Beryl W. Sprinkel today accused Japan of failing to live up to commitments to open its vast capital markets to the outside world and encourage use of the yen in international trade.

In a press conference here, Sprinkel spoke of "very troublesome signs" of violations of the spirit and letter of an agreement toward liberalization that Japan and the U.S. signed in May.

"Extensive Ministry of Finance rules and regulations will inhibit the free play of market forces" in capital markets, Sprinkel said. "We also see excessive protection of Japanese financial firms from direct competition by non-Japanese firms."

After two days of talks on progress in implementing the agreement, Sprinkel said he found "a general reluctance to allow competition to flourish in a range of areas."

Vice Minister of Finance Tomomitsu Oba, who appeared with Sprinkel at the press conference, did not respond directly to Sprinkel's charges. But he said the two countries have differences on some issues and the United States should not expect to get its way on all of them.

Sprinkel's words were the strongest from a senior U.S. official in recent months, as the two countries observed an unofficial truce in their disputes during reelection campaigns by President Reagan and Prime Minister Yasuhiro Nakasone.

They followed public expressions of concern by Japanese business executives and government officials that the Reagan administration may begin new efforts to pressure Japan to open its markets and restrict exports. These concerns arise from Japan's growing surplus in its trade with the United States, expected to reach $30 billion this year.

The United States wants its banks and companies to have access to Japan's capital markets, where low interest rates prevail. It also wants the yen to be used more in international trade, on the assumption that that would take upward pressure off the dollar and improve the balance of trade between the two countries.

The May agreement was another step in the deregulation Japan has implemented in its financial system in recent years. It was both in response to foreign pressure and some sentiment within Japan that banks had outgrown the tight controls imposed after World War II.

The May accord provides that foreign institutions will be allowed into trust banking and management of pension funds, now worth more than $60 billion. However, the Japanese argue that since only eight Japanese banks take part in this business, foreign participants should be held to that number. Sprinkel said there should be no limit.

Sprinkel also said a bankers acceptance market under development should have both yen- and dollar-denominated instruments and be open to securities houses.