U.S. and Japanese officials resumed negotiations yesterday on a knotty semiconductor trade dispute that could determine how well American high-technology companies fare in the global market.

If no agreement is reached by midnight Wednesday, U.S. trade officials said, the Reagan administration will brand Japan as an unfair trader -- a designation the government of Prime Minister Yasuhiro Nakasone is seeking to avoid through the negotiations. The United States then would penalize Japan with high tariffs and possibly quotas against imports of Japanese semiconductors and products using them.

"The ball is in their court. They have to come back to us with something," a U.S. trade official said yesterday.

Talks resumed in the afternoon at the request of officials of Japan's Ministry of International Trade and Industry, who were told Saturday by the U.S. team to return to the bargaining table when they had something to offer.

The negotiations are an attempt to settle three trade complaints brought by U.S. companies and the Reagan administration -- two charging Japanese makers with dumping chips in this country below their fair market value, and the other accusing Japan of blocking U.S. companies' sales there.

Meanwhile, representatives of industry groups from both countries have leveled new accusations, apparently to bolster their governments' stances in the already contentious trade talks.

U.S. industry officials charged Japanese companies with taking advantage of a break in the semiconductor talks to intensify their dumping of chips. And Robert Noyce, vice chairman of Intel Corp., a leading U.S. semiconductor manufacturer, accused the Japanese of reneging on earlier agreements.

For their part, Japanese companies began leaking preliminary drafts of the trade agreement that they said would more than double the cost of basic semiconductor chips that already face stiff price competition from manufacturers in other nations, especially Korea.

While the Nakasone government apparently wants a broad agreement on all the unfair trade complaints without official findings that would be a slur on Japan's reputation, Japanese companies have indicated they would rather pay the high tariffs for dumping chips than agree to an overall settlement.

The U.S. industry and the Reagan administration insist any settlement must include an end to dumping in this country, which means a price increase, guarantees of greatly increased sales of U.S. chips in Japan, and checks on Japanese global sales to make sure they aren't dumping elsewhere to gain markets from U.S. competitors.

The dispute is critical to the United States' future as a technological power because semiconductors are key elements in high-technology products, ranging from computers to advanced telecommunications systems, and are increasingly coming into everyday use in cars and household appliances.

The United States pioneered in semiconductor development, but Japan has surged ahead, according to U.S. manufacturers, because of unfair trade practices that dumped products in the United States while limiting U.S. sales in the $9 billion Japanese market.

The agreement deadline has been postponed several times as U.S. negotiators suspended the determination of the dumping cases to give the negotiations more time. But Commerce Department officials said time has run out, leaving midnight Wednesday as the deadline.

If no agreement is reached the Commerce Department will assess dumping duties that could more than double the price of Japanese chips in the United States.