High-level talks in Tokyo on opening Japan's markets to more American wood products ended abruptly yesterday, a day earlier than scheduled, "because we had a substantive disagreement on the agenda," Agriculture Undersecretary Daniel G. Amstutz said.

Amstutz returned to Washington yesterday instead of continuing with a scheduled second day of talks, the second high administration official to pull out of negotiations that arose from the decision last month by President Reagan and Prime Minister Yasuhiro Nakasone to end barriers to trade.

"We had exhausted what we could accomplish at this time. There was no reason for me to stay. In view of the lack of progress and all that is happening here on the farm issue it seemed prudent to come home," Amstutz said in a telephone interview.

He had asked for the complete elimination of tariffs on American plywood, a stance Japanese officials said surprised them and required more talks to define the issues.

Amstutz's action followed a decision by Commerce Secretary Malcolm Baldrige not to send Undersecretary Lionel H. Olmer to negotiations that were to have started Monday on opening Japan's telecommunications market until the Japanese are ready to reveal details of new regulations to be imposed April 1. As a result, those talks were postponed.

The Japanese regulations, which will determine whether American companies obtain access to the second largest telecommunications market in the world or continue to be excluded, follow the passage in December of a new law turning the government monopoly into a private company.

The Japanese Embassy in Washington offered a different version of the postponement of the telecommunications talks, denying that Japan is deliberately stalling and insisting the negotiations should continue at the undersecretary level.

"People in the streets might have the idea that Japan is delaying. That is not the approach," said Peter Sato, economic minister at the Japanese Embassy here.

"We are sitting very sincerely trying to explain what revolutionary changes Japan might have" in turning Nippon Telegraph and Telephone Corp. into a private firm. He said these are technical issues that require the participation of senior officials such as Olmer, who, he added, was urged by the Japanese to go to Tokyo for the talks.

Baldrige, however, called these "the theory and the background" that had been covered in an earlier round of talks and said he wouldn't send Olmer to Tokyo until Japan is ready to "get down to the nitty-gritty" of bargaining over the issue.

U.S. Trade Representative William E. Brock, meanwhile, backed Baldrige in comments to reporters yesterday. "I think what Mac said reflects the frustration we all feel," Brock said.

He said he was "not surprised" by Amstutz's action, and earlier told a Pacific Basin trade group that American wood and paper products are "denied a chance" to participate in Japan's markets because of protectionist barriers.

"It does get weary when Japan says, 'We have a depressed industry,' " Brock continued. "I think they should walk through the farms of Nebraska and Iowa. We have a depressed industry too."

He added that the United States' trade relations with Japan are "characterized by continued frictions.

"No nation has benefited so much and offered so little from the U.S. policies on free trade" as Japan, Brock continued.