U.S. Trade Representative William E. Brock, agreeing with senators who expressed increasing frustration over the slow pace of trade talks with Japan, said yesterday "it may be time" to take retaliatory action against countries that deny American companies equal access to their markets.

"It may be time in a number of instances to do more than talk," Brock told the Senate Foreign Relations Committee, indicating that Japan may not be the only target of U.S. trade retaliation.

He added that the Reagan administration has "begun to explore" retaliatory action, probably using new authority given the government in last year's trade act "to get more aggressive."

The problem, Brock continued, is finding retaliatory actions "that don't do us more damage than the other party. You don't hurt somebody by shooting yourself in the foot."

Brock's comments came after senators said they were turning from their long-time free-trade philosophy and beginning to believe in retaliation because of Japan's refusal to open its markets.

"No one can export manufactured goods to Japan -- not the United States, not the European Community, not the Canadians, not the OECD countries the 24 industrialized nations of the world , not the less developed countries and certainly not the world in total," said Chairman Richard Lugar (R-Ind.), who is getting the Senate Foreign Relations Committee involved in trade issues.

"I'm about at the end" after 30 years of free trade "with the absolute stonewalling by the Japanese on the opening of their trading markets," added Sen. Daniel J. Evans (R-Wash.).

Across Capitol Hill, Commerce Secretary Malcolm Baldrige heard Rep. Hal Rogers (R-Ky.) of the House Appropriations Committee's subcommittee on commerce say that pressure for market access for U.S. products "is rampant" in Congress.

Sen. John Kerry (D-Mass.) expressed "a deep-felt sense of frustration" that worsened because of indications from Brock's testimony that negotiations with Japan lead nowhere. He said it appears the Japanese welcome talks and use them to increase their market share.

"Isn't it time," asked Kerry, "for us . . . to take a stronger series of steps to make clear our determination to be able to compete on a level playing field?"

Talks are under way in four areas -- telecommunications, wood and paper products, sophisticated electronics, and pharmaceuticals and medical equipment -- as a result of an agreement by President Reagan and Prime Minister Yasuhiro Nakasone to open Japan's markets to American products.

"We are constantly told by Japan they have an open market, and it simply is not factually true," said Brock.

The real signal of Japan's intentions will come April 1, when new regulations are due that either will open the Japanese market to highly competitive U.S. products or close it even tighter than now.

Brock said Japanese markets are closed to American products in the area of wood and paper, for which Japan last week flatly refused to eliminate "substantial tariff and non-tarrif barriers." Without those barriers, low-cost products from the United States would enlarge the size of the market beyond its $2 billion by increasing demand.

Brock said U.S. negotiators have been told by Japan there is no chance of "near-term progress" in that area because of the small but politically potent Japanese industry, which demands protection.

While Brock, Baldrige and Congress may take a tough stance on trade issues with Japan, there are indications that the Reagan administration is not united. The State Department, which is running this series of trade talks with Japan, was reported by reliable administration sources to be expressing annoyance with some of the tough statements coming from trade officials.