A key congressman on trade issues yesterday accused Japan of stonewalling in semiconductor negotiations that appeared deadlocked despite the announcement last month of a "framework agreement."

"Apparently the Japanese are not serious about resolving U.S. complaints over the lack of fair access to their markets," said Rep. Sam M. Gibbons (D-Fla.), chairman of the House Ways and Means Committee's trade panel, who has devoted most of his congressional career to fighting protectionism.

The talks are going on in face of three major trade complaints brought by domestic semiconductor manufacturers and the Reagan administration. An agreement must be reached no later than Tuesday, according to Commerce Department officials, for the United States to suspend complaints that Japan dumps semiconductors in this country at prices below their fair market value. The industry, however, can withdraw its case up to a final decision by Commerce and the International Trade Commission, set for early August.

Semiconductors, or memory chips, are key elements in computers and telecommunications systems, and are critical to the nation's maintaining a lead in the new technologies. U.S. manufacturers dominated the early days of chip development, but in recent years Japan has moved ahead.

U.S. Trade Representative Clayton Yeutter, who announced the framework agreement after talks in Tokyo, remains optimistic that an agreement can be reached with the Japanese. Deputy U.S. Trade Representative Michael B. Smith held talks Monday, but they were suspended Tuesday and resumed yesterday afternoon.

Other sources, including administration trade officials and Capitol Hill trade specialists, said the talks are going badly and appear deadlocked. There are reports that Japanese negotiators, headed by Vice Minister of International Trade and Industry Kazuo Wagasugi, have backed away from promises that led Yeutter to declare that both sides had reached a framework agreement.

That could be embarrassing for the administration since President Reagan, as part of his attack on a House-passed trade bill, hailed that agreement as a White House success in forcing trading partners to play fair with U.S. industries.

In his statement yesterday, Gibbons threatened congressional action against Japan if it fails "to reach a fair agreement with us" on the semiconductor issue. He said the House-passed bill contains provisions that could be used against Japan when paired with bills awaiting Senate action.

"The Japanese must realize that the outcome of the semiconductor case is critical," Gibbons said.