The White House today will hosts a major closed-door conference to explore ways the administration can help the troubled American steel industry.

Steel industry executives and union leaders will try to convince administration officials that President Carter must work out some form of restraint on steel imports from Japan and the European Community to forestall more layoffs and mill shutdowns.

Nearly 20,000 workers have been laid off since mid-summer due to slow demand for U.S. steel products - in part because of a sluggish economy and in part because of increasing penetration of the American market by foreign steel producers.

The administration has been cool to the idea of negotiating voluntary quotas with Europe and Japan, although the Japanese government and some European producers have said they would be willing to restrict the amount of steel they send to the United States in return for for some concessions from the United States.

Industry and union leaders here have promised that if the administration does not devise some form of import relief, they will turn to Congress for help.

A steel caucus has been formed in the House and the Senate to push the case of steel industry.

Yesterday the Senate adopted a resolution calling for strict enforcement of the nation's trade laws to eliminate unfair competition from foreign steel makers.

Today's White House conference was called by Robert S. Strauss, the President's chief trade negotiator, Strauss has said that he does not believe quotas will do much to help the steel industry's problems which are rooted in outmodel plants, unwillingness to compete priceswise and lack of capital to carry out modernization projects.

The White House conference - which will bring together management, labor, legislators, importers and steel users - is designed to kick off a major federal study of policy options facing the administration.

Treasury Undersecretary Anthony P. Solomon will head the interagency task force to study the steel industry and make policy recommendations. Last week the Council of Wage and Price Stability said that the steel industry's difficulties are deep-seated and that no single government action - such as the quotas advocated by the in -dustry - could do much to solve the problems.

Steel makers have filed, or are preparing to file, many legal actions against foreign producers charging that Japanese and European steel companies are selling their product below cost in the U.S. market.

The Treasury found last month that Japan was dumping carbon steel plates and, if the dumping is found to injure American producers, special products.

The campaign for protection by the U.S. steel industry is merely the latest and most dramatic manifestation of protectionism among American workers and companies. Strauss has already negotiated voluntary import restraints in footwear and color television sets to forestall stronger restraints advocated by the industries.

In another development, the Labor Department said the 1,700 workers laid off in 1976 at two U.S. Steel Corp. plants are eligible for special federal help because they were laid off as a result of increased imports.

About 60,000 steel workers are receiving such special adjustment assistance because of import-related lay-offs, according to Lloyd McBride, president of the United Steel Workers union.