The failure of Merrill Lynch Securities to become the first foreign member of the Tokyo Stock Exchange has embarrassed Prime Minister Yasuhiro Nakasone by creating a flurry of rumors about back-door deals.

The Japanese company that ruled on Merrill Lynch's application, Yamaichi Securities, denies any unfair dealings took place.

Merrill Lynch was the first foreign company ever allowed to compete for a vacant seat on the Japanese stock market. The Merrill Lynch application was viewed by many here as a test of Japan's resolve to open its capital markets to foreign investors.

Nakasone and Yamaichi have given conflicting accounts of the highly sensitive selection process.

The vacancy was created by the amalgamation of three companies in the group headed by Yamaichi, one of the "Big Four" Japanese securities houses. Yamaichi was thus authorized to select the successor. It was the first opening in the market since 1981.

Earlier this week, Yamaichi announced it would award the seat to a comparatively small Hiroshima-based company, Utsumiya Securities Co., which is partly owned by Yamaichi.

Utsumiya had bid close to $6.7 million for the seat on the stock market, which is the world's second largest after the New York exchange. The bid topped Merrill Lynch's offer by almost $2 million, according to Japanese press reports. Seven other Japanese companies also were vying for the seat.

The decision could add to Washington's growing list of trade complaints against Japan, which is now expected to log a $35 billion trade surplus with the United States this year.

The announcement came one week before Nakasone is due to meet with President Reagan in Los Angeles. Nakasone apparently had hoped that admission of Merrill Lynch to the stock exchange would be seen as evidence of Japan's good faith toward working out frictions with the United States.

The Japanese-only rule in the 83-member Tokyo exchange had long been viewed as a symbol of protectionism by foreign governments. Currently, three Japanese securities houses are full members of the New York Stock Exchange.

Japanese officials say seats in the Tokyo exchange are limited and there is a long line of Japanese firms unable to crack the club. They argue that foreign firms are allowed to do business in the market by using full members as agents.

As part of an agreement signed with the United States in May to open its capital markets, the Japanese government pledged to work toward full membership for foreign firms. It cautioned, however, that it could not guarantee success because the stock exchange is a private-sector operation.

In a lunch with American journalists at his official residence yesterday, Nakasone termed the outcome "quite regrettable" and said further studies of reforming the system would be conducted. But he said that Merrill Lynch had been given "equal footing" with Japanese firms.

"I do not think that the American people would ask for any special privilege on such issues," Nakasone said.

The fact that higher bids had been tendered, he said, was "made known to Merrill Lynch," and the firm was given a chance to raise its offer, but declined.

The confidentiality of the bids -- or the alleged lack of it -- has emerged as a major issue in Japanese press accounts and in the securities industry since the decision was announced in a Christmas Day press conference.

Unconfirmed Japanese press reports, citing industry sources, have said that Merrill Lynch's bid was leaked to its competitors, prompting Utsumiya to raise its offer to assure a comfortable margin.

But today, Yamaichi officials told reporters that bids had been conducted in complete confidentiality. No company was told of any other company's bids and no company changed its bid, Yamaichi said.

In a telephone interview today, Tetsundo Iwakuni, chairman of Merrill Lynch's Japan operations, declined to say whether his company had been given a chance to raise its bid. But he said the firm did not want to pay an "extraordinary premium." That would set a precedent that could make it harder for other foreign companies to enter the exchange in the future and would lay his giant company open to charges of rolling over smaller ones, he said.

All applicants were required to pledge a base payment to the stock exchange computed from the size of their operations in Japan. In Merrill Lynch's case, it worked out to about $5 million. Companies were then expected to offer to pay a premium on top of the base payment, which became the key figure in the bidding.

Two companies bid higher than Utsumiya, according to industry sources, but were not selected because they were considered weaker on nonfinancial considerations. Utsumiya was strong on enthusiasm, reputation and standing, and "historical links" with the Yamaichi group, Yamaichi said.

Since Yamaichi now holds the seat, it was considered fair here that it should be able to consider its relationship with the successor in making its decision. However, the fact that the seat went to a company often considered to be part of the Yamaichi group has been widely discussed in the Japanese press.

An Utsumiya official said today that his company, the largest in Hiroshima, already does most of its business through Tokyo and a seat of its own here is a natural evolution. The membership will cost Utsumiya more than double its last fiscal year profits of about $3 million.