Nissan Motor Co. has agreed to buy a multimillion-dollar American-made supercomputer for its Japanese design facilities following negotiations that had been closely watched as a litmus test of Japan's willingness to purchase U.S. high-technology products.

The contract will go to Cray Research Inc., a Minneapolis company that is considered the world leader in supercomputers, Nissan said today in Tokyo.

The contract was awarded to Cray despite a late, high-level sales pitch by Hitachi, which holds a loose connection to Nissan management through a web of interrelationships that often governs how business is conducted in Japan.

Hitachi's attempt to get the Nissan contract surfaced here as the Reagan administration was applying pressure to Japan to open its markets to competitive American products, and Congress was urging the president by lopsided votes to take retaliatory action against Japanese protectionist trade policies.

Japanese political leaders urged Nissan to buy the Cray supercomputer to avoid worsening the trade friction between the United States and Japan.

Commerce Secretary Malcolm Baldrige put his weight behind Cray's bid when he wrote Keijiro Murata, Japan's minister of international trade and industry, in February to point out that Cray is on the "leading edge" of international technology.

Baldrige said that Nissan's decision should be made solely "on the basis of the technical merits of the equipment and software."

Nissan said it selected Cray after "an exhaustive evaluation" of Japanese and U.S. products that showed the American-made computer was "best suited" to its needs.

"We're delighted," said Frank Brett Berlin, Cray's Washington representative. "The thing that really delighted us is that the technical evaluation put us on top. It was a direct head-to-head competition with a Japanese manufacturer in an environment that would have given any benefit of the doubt to the Japanese.

"That's the real sweetness of victory. The Nissan technical people really ran us through the wringer."

Cray has continued trying to sell supercomputers to Japanese companies after giving up on on efforts to sell to the government or institutions such as universities that are supported with government funds because of what appeared to be a ban on the purchase of the American product.

In a number of cases, universities wanted to buy a Cray for advanced research, but were forced by the Ministry of Education to use less-sophisticated Japanese-made products.

Japan now is in an international race with the United States to produce supercomputers, which can The Nissan technical people really ran us through the wringer." Cray official Frank Berlin handle highly complex problems in a short time. They are widely used around the world for designing cars, planes, weapons and nuclear plants, as well as creating the animated effects in science fiction movies.

Nissan said it needs a supercomputer "to help work out sophisticated design computations and ever-complicated engineering problems" on its cars. It also said the Cray supercomputer will "prove indispensable" in the aerospace field, where the company is expanding its activities.

Cray computers are used by major American and European car makers, athough the Nissan purchase is the first by a Japanese manufacturer. Because of competitive presures in the industry, Cray hopes that other Japanese car makers will follow suit.

A Cray supercomputer costs between $5 million and $20 million, depending on the extras. It was unclear last night how much the Nissan purchase would be worth.

Congressional staff aides who specialize in trade questioned last night whether Nissan's purchase of a Cray supercomputer is an indication of a further opening of the Japanese market or merely a one-time event caused by the intensified pressures on Japan to buy more competitive American products.

They recalled that in 1983 Toyota refused to buy a $3 million American telephone system, even though it was considered more sophisticated than the domestic product Toyota bought.