President Reagan dispatched a personal envoy to Tokyo last night to meet with Prime Minister Yasuhiro Nakasone in a last-ditch effort to resolve a trade dispute that White House sources said threatens to escalate into anti-Japanese protectionist actions by Congress.

The president acted after being told that Japanese negotiators have not kept the commitment that Nakasone gave him in January that Japan's newly denationalized telecommunications market will be as open to American products as the U.S. market is to the Japanese.

On Monday new Japanese regulations take effect that can either assure American companies of equal access or keep the door shut against them.

"We've made some progress," but "it's not enough," said an informed administration official.

The special presidential envoy is Dr. Gaston Sigur, a Japanese specialist on the staff of the National Security Council who has a long-standing personal relationship with Nakasone, sources said. His trip was supposed to be a secret mission, administration sources said.

The Sigur mission is the cornerstone of an "intensified" U.S. effort over the weekend to settle the telecommunications issue. The attempt originated with the Senior Intergovernmental Group on International Economics, a Cabinet-level committee that heard a report on the status of the negotiations from Commerce Undersecretary Lionel H. Olmer and Deputy U.S. Trade Representative Michael B. Smith.

If the issue is not settled satisfactorily, administration officials fear it could undermine relations between the United States and Japan, its closest Pacific ally. The issue could persist through the May economic summit in Bonn, an administration source said.

"This is a moment of truth for them the Japanese ," added the source, who is familiar with the Cabinet-level discussions.

At that meeting, sources said, "there was a feeling that if the White House declares defeat, Congress will go up in smoke" and pass legislation by an overwhelming -- perhaps veto-proof -- majority lashing out at Japan and possibly hurting U.S. interests in the process.

"If the White House declares a victory," the source continued, "no one will believe them," especially since a "scorecard" listing the administration's objectives in the talks is in wide circulation here and in Tokyo.

The telecommunications talks have become the symbol of what is perceived here as unfair trading tactics by a nation using its friendship with the United States to flood this country with its goods while protecting the Japanese market from competitive U.S. products.

The Senate, by a 92 to 0 vote, delivered a rare rebuke to a key ally Thursday by calling for trade retaliation against Japan for its protectionist policies.

The White House and State Department responded in a low-key manner to the Senate action, refusing to condemn it as they have earlier protectionist moves while emphasizing the seriousness with which the administration views the telecommunications talks.

"Japan must rapidly remove barriers to its markets. That is a matter of the highest priority to us," said Ed Djerejian, a State Department spokesman.

White House spokesman Larry Speakes said "as a general policy" the administration opposes protectionist legislation.

Olmer, a key U.S. negotiator on the telecommunication talks, told a House subcommittee Thursday that the United States had won full agreement from the Japanese on two of nine key issues. Later, sources said, Japanese Vice Minister of Post and Telecommunications Moriya Koyama had been more forthcoming as the talks continued, but had not gone far enough to satisfy the U.S. side. Koyama left for Tokyo yesterday.

At issue are the complex bundle of regulations that give Japanese bureaucrats unlimited authority to certify products for sale in Japan. The Reagan administration is fighting to pare down these regulations, give foreign suppliers a voice in making the rules and establish an appeals process against bureaucratic power.

In the process, the administration has demanded the same access for American companies to the Japanese market as Japanese companies receive in the United States.