Commerce Undersecretary Lionel H. Olmer pointed yesterday to "encouraging signs" in trade talks aimed at opening Japanese markets to American manufactured goods but warned Congress that barriers remain to increased U.S. sales.

"In the past year and a half, we have made remarkable progress," he told the House Banking Committee's economic stabilization subcommittee. "Our telecommunications agreement, even though it does not yet deal with all the market-opening measures necessary for our entire industry, is nonetheless a milestone."

He also called "historic" Prime Minister Yasuhiro Nakasone's agreement to hold negotiations that could change whole sectors of the Japanese economy and said the prime minister's "personal stake in this process is unprecedented."

Olmer, who plans to leave his job in early June, has headed American negotiating teams in telecommunications talks. He also cited progress in two of the three other areas that will be covered in market-opening talks in the January meeting between President Reagan and Nakasone. They are medical equipment, where he said "inroads have been achieved" on Japan's previous blanket refusal to accept foreign test data, and sophisticated electronics, where several major barriers already have been dealt with.

He also credited congressional pressure on Japan for helping to win trade concessions, but suggested that Congress hold off on legislation that would restrict imports from Japan in retaliation for that country's protectionist policies.

"It would be wrong now to promote passage of drastically protectionist legislation right after the Japanese government has done almost everything we asked of them" in the first phase of telecommunication trade talks, Olmer told reporters after his House testimony.

He will be testifying Friday on a bill introduced by Sen. John C. Danforth (R-Mo.) that would restrict imports of Japanese telecommunications equipment if U.S. companies are prevented from selling in Japan. Olmer had given his conditional support to that bill in earlier testimony, but the Reagan administration has not formulated its position.

Olmer detailed new areas for new telecommunications talks in a letter sent Friday to his Japanese counterpart, Vice Minister of Post and Telecommunications Moriya Koyama. Among them are changes in Japan's radio wave laws "which make it difficult for U.S. firms to do business in Japan," Olmer wrote Koyama.

"There are many specifications in Japan which are different from specifications in the rest of the world, not better -- just different. This requires unique product development just for the Japanese market," Olmer said in the letter. Because that market is "significantly smaller" than the American market, it offers "less economic incentives" for U.S. companies to meet Japanese standards than the incentives Japanese companies have to modify equipment to take advantage of America's far larger market, he noted.

He also wrote Koyama that Japan's radio-wave law excludes sophisticated, state-of-the-art equipment because Japanese companies do not manufacture such equipment.