Congress yesterday intensified its pressure on the Reagan administration to punish Japan for protectionist trade policies, but the State Department warned that "retaliatory moves could hurt us just as much as the Japanese."

The House passed a nonbinding resolution by an overwhelming 394-to-19 vote urging the president to take retaliatory moves against Japan while, earlier, the Senate Finance Committee approved a bill that would require the president to take steps against Japan. The bill was a stronger version of a nonbinding resolution that sailed through the full Senate by a 92-to-0 vote last Thursday.

In Tokyo, Japanese Foreign Minister Shintaro Abe described the Senate committee action as "very regrettable," Washington Post correspondent John Burgess reported last night. "It would not only discriminate against Japan, but threaten world trade as a whole," Abe said. A statement issued by the Foreign Ministry said Congress should recognize that Japan is working hard to open its market.

In Washington, the Reagan administration, looking over its shoulder at Capitol Hill, insisted that U.S. negotiators are making progress in trade talks with the Japanese. This appeared to be an attempt to blunt increased congressional and public concerns over the United States' $36.8 billion trade deficit with Japan and that country's unwillingness to open its markets to competitive American products.

In a day filled with trade activities, the White House also announced that President Reagan will nominate Clayton Yeutter as the new U.S. trade representative. Yeutter, who served as a deputy special trade representative in the Ford administration, will succeed William E. Brock, who two weeks ago was picked by Reagan to be labor secretary.

"Our trade policy toward Japan is clear," the administration said in a statement released by the State Department. "We want the same access to Japan's markets that Japanese companies have to ours. Japan needs to open its markets fully to American products and services.

The statement also praised as a "welcomed development" the progress in trade talks that resulted from an agreement between Reagan and Prime Minister Yasuhiro Nakasone last January for negotiations to open Japanese markets.

The House moved speedily on its resolution pressing for presidential moves against Japan, bringing it to the floor last night just hours after it was reported out by the House Ways and Means Committee.

Even such free traders as Rep. Sam Gibbons (D-Fla.), chairman of the Ways and Means Committee's trade subcommittee, voted in favor of the resolution. Only two Democrats and 17 Republicans voted against the measure.

The House resolution, however, was more tempered than the Senate measure passed last week, mentioning the budget deficit and strong dollar as well as closed Japanese markets as causes of the United States' record trade deficit. This led to criticism by some House members, who said they preferred sending a stronger message to the Japanese and the White House.

Earlier, the Senate Finance Committee, in an effort to turn up the heat on the Reagan administration and on Japan, approved the nonbinding resolution passed by the Senate last week as a bill that could force presidential action.

It appeared, however, that the Senate leadership was content to let the bill stand as a club against Japan rather than force the president to either veto the measure or accept it. Senate Majority Leader Robert J. Dole (R-Kan.), who voted for the bill in the Finance Committee, said he "will not bring it up for a while" and hopes members "won't press it."

The bill, moreover, cannot be brought up on the Senate floor without being attached to a House measure. Trade bills are considered revenue measures and, as such, must originate in the House.

Dole added, however, that "the president might want to refer to it" during the May economic summit in Bonn, where there are indications that Japan may be pressed in private talks to open its markets to all nations in the world, not just the United States.

While legislators were debating trade on Capitol Hill, Reagan's senior economic advisers at a White House meeting attempted to analyze Japan's latest response to U.S. efforts to open its newly denationalized telecommunications industry to American products.

According to administration sources, the Cabinet-level meeting kept a watch on Capitol Hill as it tried to come up with a public statement encouraging Japan to continue liberalizing its trade policies without adding fuel to growing anti-Japanese sentiment.

The meeting was described by informed administration and congressional sources as so touchy that a scheduled press briefing by four officials -- Commerce Undersecretary Lionel H. Olmer, Undersecretary of State Allen Wallis, Deputy U.S. Trade Representative Michael Smith and Assistant Treasury Secretary David Mulford -- was abruptly canceled in favor of a two-page, single-spaced statement issued by the State Department.

The statement said the telecommunications trade talks reached "basic understandings" that stripped away some institutional barriers to U.S. suppliers. These included two key points that Nakasone agreed to over the weekend in talks with special envoys Reagan sent to Tokyo -- National Security Council aide Gaston Sigur, a longtime friend of the prime minister's, and Olmer.

But other barriers remain, and the administration said it will continue monitoring the new understandings and "continue to pursue further market opening actions to remove formal and informal barriers" in telecommunications.

Talks are continuing, the administration said, on the three other sectors that Nakasone and Reagan agreed should be further opened to U.S. suppliers: forest products, sophisticated electronics and pharmaceuticals and medical equipment.

It stressed the close strategic and economic ties that have developed between the United States and Japan, emphasizing that Reagan has said "there is no relationship more important for peace and prosperity in the world than that between our two countries."