Powerful Japanese business interests are moving toward an era of expanded trade with North Korea in what may be the beginning of a new open-door policy toward that long-closed communist nation.

Some of Japan's most prominent industrialists and bankers have put together a new trade research organization that hopes to tap North Korea's rich mineral resources and enlarge substantially the two-way trade with the Asian neighbor.

Its formation is one sign of a growing Japanese interest in creating closer ties with North Korea, a country considered out of bonds until recently because of Japan's friendly relations with South Korea.

The move by leading businessmen and a recent visit to Pyongyang by conservative members of the Japanese parliament have alarmed and irritated South Korea, which regards any friendly country's interest in the communist enemy as a stab in the back. There have been hints that South Korea might retaliate with boycotts against Japanese firms that deal with the North.

The key to the new trade overtures is the prominence of the businessmen who have formed the vaguely named East Asian Trade Research Board here. Among them are Yoshihiro Inayama, chairman of Nippon Steel Co. and head of the powerful Japan Federation of Business Organizations: Sohei Nakayama, counselor of the Industrial Bank of Japan, and Tasuzo Mizukami, chairman of the Japan Foreign Trade Committee and former head of a leading trading company.

Executives of Japan's six major trading companies, a television network, and several banks, as well as members ne organization. All are influential in the business community and many have important ties with the government.

The Research Board is also said to be interested in studying trade with Taiwan, but its major focus is on creating new trade relationships with North Korea.

Its member have avoided publicity, for the reason that it would attract antipathy from South Korea, but a general outline of its purpose was provided during interviews in business circles here.

The board is said to believe that North Korea is abandoning its do-it-yourself style of economic modernization and is preparing to seek more assistance and trade opportunities with the outside world. The govenment of North Korean President Kim II Sung would like to minimize economic ties with the Soviet Union because of the political conditions that come with them and will turn more and more to noncommunist countries.

Japan should be prepared to take advantage of such an opening, these sources contend, particularly now that North Korea has partially recovered from the recession of the mid-1970s and is starting to pay off its foreign trade debts.

"Since their debts are now being paid, we think that North Korea is becoming more serious," one source knowledgeable about the new organization's intentions said.

The organization does not want to destroy business relations with South Korea, the source added, "but we cannot deny that we want to build a bridge between Japan and North Korea."

Until now, trade between North Korea and Japan has been a trickle, managed by a number of smaller companies on an unofficial basis through dummy trading organizations.

That trickle was reduced even more after North Korea, suffering from a recession induced by high oil prices in the mid-1970s, failed to pay about $400 million in debts to Japanese firms. The debts were renegotiated last year with North Korea being permitted to pay them off in installments over a 10-year period. The first two installments have been met.

Some businessmen and a member of parliament who visited Pyongyang, Katsushi Fujii, have pressed for a more formal normalization of trade through a memorandum agreement that would be signed by the North Korean government and a consortium of Japanese companies..

The model for that agreement was one signed in 1962 with China at a time long before Japan normalized relations with that country. It opened the doors for substantial trade and economic exchanges during a period when there were no government-to-government relations. Observers here see the same pattern now developing between Japanese firms and North Korea.

South Korea has already signaled its displeasure with the proposed trading arrangements. Its embassy here has attempted to find out what the Research Board is up to and has warned that any kind of trade with North Korea would increase the communist country's military power.

News dispatches from Seoul recently reported that the South Korean government will urge companies to stop dealing with Japanese companies that trade with North Korea. However, that would be a largely empty threat if made against companies of the size and importance of those now considering trading with the North.

Japan is reportedly interested in supplying quantities of manufactured goods to North Korea, as it does to many other developing countries. One of the major goals, however, appears to be to tap the large quantities of mineral resources known to be found in the North. According to informed sources, Nippon Steel, this country's largest steel company, is interested in gaining access to North Korea's particularly rich iron ore deposits as well as other metals.