Although Japan's formal trade barriers are being dismantled, foreign businessmen still find formidable obstacles to selling their merchandise in this rich market.

One by one, most of Japan's protectionist tariffs and quotas are disappearing, but the foreign seller encounters a maze of bureaucratic rules, commercial testing requirements and cultural inhibitions that are often just as difficult to penetrate.

Foreign tobacco companies can sell cigarettes in only 14,000 of this country's 250,000 tobacco outlets. Government procurement purchases have opened a potential $10 billion market to foreign sellers, who nevertheless complain the bidding rules are stacked against them. It's difficult to find out which cosmetic substances the government bureaucracy will permit in the market.

And beyond these so-called "nontariff barriers" lie the cultural facts of economic life in Japan: longstanding business ties, often of an almost familial closeness, lead Japanese buyers to deal exclusively with Japanese suppliers.

Yet many successful American businessmen here believe the market is open to the entrepreneur who tries hard enough and that the major obstacle is the failure of U.S. corporations to make the long-term effort needed.

The time when Japan "rigged the rules" against foreign sellers is passing, Mark A. Zimmerman, president of the American Chamber of Commerce in Japan, said in a recent speech. "I am confindent that American business can garner a significant share of the Japanese market if our corporations are prepared to make the same level of commitment per capita that they make to the U.S. and Canadian and European markets."

Zimmerman observed in a interview that U.S. corporate investment is only one-tenth the size of that in the United Kingdom, even though the returns on investment here average an annual 18 percent -- enough to whet the appetites of any business person.

"American companies that have been here more than 10 years and have good local employes are the ones which succeed," Zimmerman added. "The others -- what I call the toe-in-the-water bunch -- don't."

"You can sell to the Japanese, no matter what, if you have the better mousetrap," asserts William Kyle, a businessman whose new company acts as go-between for American manufacturers and government agencies here. "But you've got to be able to service what you sell. If something goes wrong, the Japanese company expects you to have a service rep in their office within two hours. Not many American companies offer that."

Both sides of the argument about Japanese imports are apparent in the far-reaching agreement to open up Japanese government procurement to foreign suppliers. It centers on the $3 billion market in products bought annually by Nippon Telephone and Telegraph.

Since the agreement went into effect in January, the company has abided by the rules and advertised bids for foreigners. But American businessmen complain the bids close after 30 days, not long enough to prepare an offer. A Houston company received an ad for computer equipment two weeks after it was offered, took one week to prepare its bid, and mailed it in. It arrived one day after the bids closed.

The other side of that coins is that few American companies competent to bid on sophisticated electronic equipment have offices here. So far as could be learned, no U.S. company has opened an office here since January specifically to take advantage of the new market.

Long a citadel of protectionism, Japan, under pressure from foreign governments, began dismantling tariffs and quotas in the 1970s. When the new agreements reached under the Tokyo Round of trade negotiations are in place, Japan's market will be technically as open to foreign sellers as the American and European markets.

Agricultural commodities are an exception. Japan's market is still studded with quotas and tariffs on farm products, reflecting the power Japanese farmers wield in politics here. Finished leather goods also are protected strongly.

But many businessmen insist that Japan imposes excessive testing and safety rules that have the effect of discouraging foreign companies from entering the market.

For example, American automobile companies insist that the price of their vehicles is inflated beyond the competitive edge by Japan's strict safety requirements. Japan requires a special heat-shielding device around every car's catalytic converter. Americans say it isn't necessary. Japanese say it is needed to prevent heat generated at high speeds from causing brush fires in rural areas.

Needed or not, the heat shield adds about $250 to the price of each Chrysler brought into Japan, a company spokesman says. It costs only $45 on a Japanese car because it is installed in large volumes in Japanese factories.

Other impediments seem more blatant. Although American tobacco is eagerly sought here, the Japan Tobacco and Salt Monopoly, a government corporation, has a vested interest in keeping it out. The monopoly insists that each outlet selling foreign tobacco must have a certain amount of capital and must be located where there is a certain amount of daily passersby as potential customers. It also must order at least 30 cartons of cigarettes a month from foreign suppliers.

The monopoly argues that such rules are needed to protect the outlet from going bankrupt. Such paternalism has the effect of permitting only about 14,000 outlets to deal in foreign tobacco. Under foreign pressure, the monopoly has agreed to bend the rules "flexibly" to permit about 20,000 outlets to handle foreign tobacco by the end of this year.

Gradually, some of the barriers have been worn down by persistent outside pressure over the years. Zimmerman cites a case in the pharmaceutical industry in which, at one time, foreign companies had only one year of protection from competitors when they submitted new products for testing and monitoring in Japan. Japanese companies enjoyed three years of protection for newly licensed drugs. Under pressure, the Ministry of Health and Welfare agreed to make it six years for both foreign and domestic producers.

Amid falling tariffs, fading quotas and persistent outside pressures, Japan in the '70s increased substantially its imports of manufactured goods. They more than doubled between 1970 and 1979. But in 1980, they actually declined by about 2 1/2 percent from the year before, a retreat Japanese attribute to slumping demand here.

Even after a decade of change, Japan remains far behind Western industrial nations in terms of manufactured imports as a proportion of total imports. It imports large quantities or oil and other raw materials but comparatively small amounts of foreign manufactures. Last year, only 22 percent of its imports were in manufactured goods.

That low level is a major factor in the trade deficits that Western countries suffer with Japan. The U.S. had a $10 billion deficit with Japan last year. Europe's was $11 billion.

Those deficits have prompted both Europe and the United States to demand that Japan open up its market, and there have been blunt warnings from some European countries that Japan must open the door wider or face spreading, overt protectionism.

Some observers believe residual cultural factors in Japan make it impossible to open this market further. Big Japanese companies that control markets like the security of dealing with long-standing suppliers and their own affiliates, and would continue to buy Japanese goods even if foreign firms offered better products and lower prices they say.

Jorn Keck of the European Community Delegation Office here believes if all tariffs and quotas were abandoned or equalized, and if all nontariff barriers were negotiated away, the cultural barriers would continue to stymie foreign companies. He suggests that the Japanese government go beyond passive openness and require Japanese companies to buy abroad.