U.S. trade negotiators today condemned plans by Japanese semiconductor producers to expand their plant capacity, saying it would worsen a world oversupply of the devices and encourage dumping.

The statements, voiced at an inconclusive five-hour meeting with Japanese officials here, came as the U.S. government is investigating a formal complaint of unfair practices by the Japanese industry in its trade with the United States. The complaint was filed by U.S. semiconductor manufacturers.

"We asked the Japanese government to take a clear public position that the industry will not be bailed out if it gets into trouble" because of overcapacity, said Clyde Prestowitz, counselor to the U.S. secretary of Commerce and a member of the negotiating team. The team was led by acting U.S. Trade Representative Michael Smith.

According to other accounts of the meeting, Japanese officials responded today by saying that companies here already are reducing their investment plans because of a downturn in the market. The Japanese officials denied that Japanese semiconductors have been dumped, or sold below cost, in the United States.

Reducing Japan's growing surplus in semiconductor trade with the United States is a prime goal for Washington in ongoing trade negotiations between the two countries. Japan last year had a total trade surplus of $37 billion with the United States.

Semiconductors are circuitry chips that serve as building blocks in computers and other electronic equipment. The United States, with semiconductor sales of about $12 billion in 1984, remains the world's leader, but is increasingly wary of being displaced by Japan. Japanese production in 1984 totaled about $8 billion.

Japan's U.S. semiconductor sales grew by about 200 percent last year to $1.5 billion, according to the Electronic Industries Association of Japan. Market share has grown to the vicinity of 17 percent, by U.S. accounts, though some Japanese put it nearer 14 percent.

U.S. sales of semiconductors here totaled about $650 million last year and grew more slowly, by about 150 percent, the Japanese association said. U.S. officials say market share has been declining in recent months and now stands at about 10 or 12 percent. Again, the Japanese dispute these figures, saying they rise much higher when production by U.S. subsidiaries in Japan and imports from U.S. companies producing in foreign countries are added.

The U.S. requests today were given added psychological weight by the unfair-practices complaint filed last week against the Japanese industry by the Semiconductor Industry Association of the United States.

U.S. negotiators insisted today there was no link between the complaint, which has yet to be formally ruled on, and the U.S. government position. But Japanese who asked not to be named spoke of them as if they were part of the same package of demands.

In its complaint, the SIA alleged that U.S. sales in Japan are inhibited by a cartel-like system operated by the major Japanese producers. It called for measures to restrict U.S. imports if negotiations with Japan do not produce satisfactory results.

U.S. officials have cited a letter sent to some U.S. sales agents of Hitachi, a major Japanese chip producer, by the company's San Jose office. It instructed them to undercut prices of competitors by 10 percent. Hitachi has said that the letter was written by a low-ranking employe and did not represent company policy.

Plunging prices for chips on the world's markets already have put a serious strain on the U.S. industry. Commerce Department officials charge that Japanese companies' plans for new plants will push prices down further and compound American producers' financial troubles.

Former Commerce undersecretary Lionel Olmer told foreign reporters here recently that the United States wanted to work with Japan to bring about an orderly reining in of plans for extra capacity. But the American team today apparently did not make so direct a request.

Japanese officials maintain that capacity decisions are up to the private sector, not the government, but that market forces are doing the job. "Japanese investors are planning considerable cutbacks in investment," said a Japanese here who follows the industry.

In today's negotiations, the Japanese also denied allegations of collusion to keep down American sales here. "There is no sort of illegal practice in Japan to bar imports of semiconductors from the U.S.," the Japanese source said.

Japanese contend that the market shares reflect the strong standing of the Japanese industry in general.

Negotiators today also discussed the possibility of tariff reductions on electronics imports. Japan has proposed mutual cuts; the United States, however, feeling that such a move might just raise Japanese exports further, prefers unilateral reduction.

Today's talks also touched on a collection of other U.S. concerns on trade with Japan, including its patent system, protection for computer software and sale of foreign satellites here.

The Japanese said they planned to streamline the patent system. Its laborious approvals process has been blamed for preventing foreign companies from making returns on new products. Prestowitz said this represented major progress.

In the next week, U.S. and Japanese negotiators also will be discussing telecommunications equipment, drugs and medical devices, and liberalization of Japan's financial markets.