The deal looked set. For the first time in four years, Cray Research Inc. appeared to have sold a supercomputer in Japan. The prospective buyer, Osaka University, was so pleased it even included a picture of the state-of-the-art computer in its catalogue.

Then the powerful hand of the Japanese bureaucracy intervened, according to U.S. officials and Cray executives familiar with the unsuccessful 1983 sales effort.

The Ministry of Education turned down the university's budget request, saying it could not buy a supercomputer from Cray but had to tie into a less-sophisticated Japanese-made Fujitsu computer being purchased for another university.

"It was not very subtle," said Cray's Washington representative, Frank Brett Berlin. "Basically, when it got to the Ministry of Education there was no budget. It's been kind of a pattern.

"We came to the conclusion that as a company we will not be able to sell to the vast majority of government-supported Japanese institutions. It's a business decision. We accept it as a fact."

The experience of Cray -- a recognized leader in high technology whose supercomputers are used around the world for designing cars, planes, weapons and nuclear plants -- provides a graphic illustration of the problems top American companies have in selling their products in Japan, one that has been repeated in dozens of other cases.

The commonly held view that Japan closes its markets to American products while flooding this country with its goods bears major responsibility for the sudden new wave of anti-Japanese sentiment in Washington. This feeling took its most tangible form Thursday when the Senate, in one of the most stinging congressional rebukes handed to a close ally, voted 92 to 0 for trade retaliation against Japan because of what Congress considers its protectionist policies.

That view, moreover, underlies the strong push by the Reagan administration to get the government of Prime Minister Yasuhiro Nakasone to remove restrictions and unofficial barriers to the import of American products, especially those that clearly are in the forefront of technological development.

These products include space satellites, supercomputers, sophisticated telecommunications equipment and drugs and medical equipment. All of these are important, emerging industries and Japan is accused of protecting and nurturing its companies in these fields until they can become world leaders.

"The consequences of exclusion from the Japanese market are severe and cumulative," said Michael Borrus, deputy director of the University of California's Berkeley Roundtable on the International Economy, in congressional testimony on the U.S. telecommunications industry.

He said the harm to American companies goes far beyond the loss of profits and the inability to stay abreast of technological change in Japan. "The absence of sustained foreign competition there in Japan has given Japanese companies a critical advantage in international competition" by allowing them to use their protected home market as a springboard for achieving a dominant position overseas, he said.

The Reagan administration's initial market-opening effort is focused on telecommunications, with the U.S. aiming to make sure that regulations due Monday for Japan's newly-denationalized industry allow American firms to have the same opportunities to sell in Japan that Japanese companies have in the United States.

It remains unclear whether the current U.S. pressure will bear fruit. After a Cabinet-level review Friday, the White House decided the Japanese had not gone far enough and President Reagan dispatched a personal envoy to Nakasone to press for greater changes.

No matter how much the U.S. effort pries open the Japanese market, it already has helped a small Waltham, Mass., company, Concord Data Systems, get an advanced piece of telecommunications equipment approved for sale in Japan after it had been initially rejected without a reason.

The rejection came in the midst of the talks. When it was brought to the attention of Donald S. Abelson of the Office of the U.S. Trade representative, it quickly was cited as an instant case study of how Japan's complex system of licensing and regulations works to keep out the most advanced U.S. products.

According to U.S. officials, the arbitrary nature of the rejection embarrassed the Japanese, undercutting claims by government officials and leading industrialists that high quality imports are welcomed in Japan.

But, said Commerce Undersecretary Lionel H. Olmer, the U.S. government cannot step in every time an American exporter runs into a stone wall in Japan. He noted that it took more than two years of intense government efforts to get Japan to accept U.S. softball bats -- all for 400 sales once the market was opened.

Although a small company, Concord Data is no stranger to international business. It has sold its autodial modem -- a device that allows computers to exchange data at high speeds over ordinary phone lines -- in 40 countries and knows how to tailor its product to meet national requirements.

In the case of Japan, however, neither it nor its Japanese distributer could find out the reason the autodial modem was rejected.

"We thought they were being very, very inflexible," said Steve Puchkoff, a Concord Data executive. "We always knew we could sell there and our distributer knew we could sell there if we could only get approval."

He anticipates a million dollars in sales a year, hardly a blip in Japan's $36.8 billion 1984 trade surplus with the United States.

Sales of Cray supercomputers are another matter, though, with each one selling for between $5 million and $20 million, depending on extras.

So far, Cray has managed to sell four supercomputers in Japan. Two are used for time sharing and one, sold last year, is in full use by Japan's telephone company, Nippon Telegraph and Telephone (NTT). The owner of the fourth doesn't want its name used.

Somewhere in the midst of Cray's sales effort, Japan decided to make a national effort to develop supercomputers of its own, with Fujitsu and Hitatchi picked to lead the charge. From then on, Cray's sales efforts stalled. Despite campaigns at Japan's leading universities and research institutions, the Osaka experience was repeated.

According to knowledgable U.S. trade sources, the NTT sale most likely had political overtones, part of Japan's efforts to buy sophisticated American products, even though the computer is more than proving its value to the Japanese company, U.S. sources say.

Cray's most recent problems revolve around its efforts to sell a supercomputer to Nissan Motors, where it appeared the deal was set until Hitachi made a late, high-level sales push. Berlin, the Cray executive, said his company is not concerned by the competition as long as Nissan makes its selection on the basis of merit, not out of a "Buy Japanese" motive.

That sale, however, has become another litmus test of Japan's intention to be a fair trader. A Japanese political leader, Yoshiro Hayashi, here for the Reagan inaugural this year, said at a private luncheon that Nissan, with all its sales in the United States, should buy the Cray computer.

And Commerce Secretary Malcolm Baldrige wrote Japan's Minister of International Trade and Industry Keijiro Murata this month pointing out that Cray's supercomputer is on the "leading edge" of international technology.

"Nissan's decision regarding its supercomputer needs and the competitiveness of Cray should strictly be made on the basis of the technical merits of the equipment and software," Baldrige wrote his Japanese counterpart.