The Senate, by a vote of 92 to 0, called on President Reagan yesterday to retaliate against Japanese protectionism.

The vote reflected growing frustration in Congress with Japan's refusal to buy American products while flooding the United States with its goods. It came as a top U.S. negotiator reported little progress in last-minute talks to open Japan's telecommunications market, and as the Commerce Department reported that the U.S. merchandise trade deficit accelerated last month.

Meanwhile, the Japanese government brushed aside U.S. criticism and formally announced a new 2.3-million-unit ceiling on auto exports to the United States.

Makoto Kuroda, director general of the International Trade Policy Bureau, said that Japan was "prepared to accept . . . criticism" from Washington that the new limit was too high. Japanese auto manufacturers condemned the new ceiling as too restrictive, however, and said it contributes to the worldwide spread of protectionism.

The Senate, charging "unfair trade practices" by Japan, urged the president in a non-binding resolution to restrict Japanese imports in areas such as cars, electronics and telecommunications unless that nation gives competitive American products free access to its market.

"Bo-eki-sen," said Sen. Max Baucus (D-Mont.), using the Japanese word for trade war during the debate on the issue.

The only opposition to the resolution came from senators who wanted even stronger action. However, the Senate was blocked by a technicality from passing a bill that would have forced presidential action: Trade legislation is considered a revenue bill that must originate in the House of Representatives.

"The best we can do is a sense of Congress resolution," explained Sen. John C. Danforth (R-Mo.), who sponsored the measure with Sen. David L. Boren (D-Okla.).

The hour-long Senate debate was filled with sharp attacks on Japan's trade tactics.

"I have reached the limit of my patience with Japan," said Sen. Robert Packwood (R-Ore.) "I am going to do anything I can to retaliate against any of their products that come into the country."

If nothing is done to solve the United States' trade problems with Japan, promised Senate Majority Leader Robert J. Dole (R-Kan.), "we'll be back here very soon with not just a resolution, but legislation."

Reagan administration officials have declined to say whether the president would veto legislation calling for retaliation against Japan, though Undersecretary of Commerce Lionel H. Olmer said yesterday, "I do not at this point accept the necessity of retaliatory measures or their desirability."

The Commerce Department said the merchandise trade deficit swelled to $11.4 billion last month, its worst showing since the $11.5 billion deficit of last September, as exports dropped 7.7 percent.

Japan's trade surplus with the United States reached $4.2 billion in February -- the highest level since last July's $4.7 billion gap.

Olmer told a House Energy and Commerce subcommittee there has been little progress in talks to get Japan to agree to regulations that will give American telecommunications products the same ease of entry as Japanese products have to this country.

Under sharp questioning by committee chairman John Dingell (D-Mich.), Olmer said the Japanese have agreed fully to only two of nine U.S. objectives designed to assure U.S. access in Japan's telecommunications market.

The trade talks on new regulations for Japan's newly denationalized telecommunications industry ended here last night, administration sources said, and Olmer and Deputy U.S. Trade Representative Michael Smith will report on their progress today at a Cabinet-level White House meeting. The sources said the Japanese gave in on some points, but were unsure whether they went far enough. Olmer said he asked Japanese Vice Minister of Post and Telecommunications Moriya Koyama to postpone his return to Tokyo, scheduled for yesterday morning, so the talks could continue.

If Koyama had gone home as scheduled, Olmer said, the talks would have had an "unsatisfactory result."

The negotiations arose from an agreement last January between President Reagan and Prime Minister Yasuhiro Nakasone to open Japan's markets to American products in four areas -- telecommunications; sophisticated electronics; wood and paper products; and pharmaceuticals and medical equipment.

Because of the new regulations taking effect Monday, the telecommunications talks were put on a fast track.

"If we are successful in our negotiations," Olmer told the subcommittee, "it will make a tremendous difference in the opportunities available to American companies."

The congressmen, nonetheless, remained unimpressed, insisting that previous Japanese market-opening agreements had not resulted in increased sales for U.S. companies because Japan raised new import barriers as fast as old ones were torn down.

Olmer replied that it will take "eternal vigilance" on the part of U.S. officials and companies to make sure Japan keeps its obligation to allow access to its market.