The Carter administration ran into a new burst of congressional demands yesterday for import restrictions to protect the nation's ailing steel industry - and a warning that if the White House does not act soon, Congress may move on its own.

At a hearing before a House Ways and Means subcommittee, panel members were almost unanimous in calling for imposition of quotas or other restrictions. Some members announced plans to form a new "steel caucus" to push through quotas legislation.

Meanwhile, the United States Steel Corp., the nation's largest steel producer, filed a petition with the Treasury Department charging six major Japanese steel firms with dumping - selling here below the prices charged in their home markets - against trade law.

If officials decide in the company's favor, it could result in imposition of countervailing duties, or compensatory tariffs. The complaint alleges unfair sales practices in six major categories of steel, mostly heavy structural products such as plates and pipes.

The intensified action served as a signal to Robert S. Strauss, the President's special trade representative, to toughen his efforts on steel in the upcoming world trade talks. The trade chief left late yesterday for a new round of discussions with the Europeans.

Strauss, who put in a brief appearance at the hearing before leaving for Brussels, conceded the administration has no "identifiable policy" for dealing with the steel industry's complaints, but warned that imposition of quotas would only invite retaliation.

"I don't believe quotas are the answer - not the short-term solution and not the long-term solution," Strauss told the subcommittee members. "I have no clear, definitive answers," he said, "but I don't think this problem lends itself to very simple . . . action."

The U.S. currently is trying to push through special multi-lateral talks on steel as part of the worldwide trade negotiations. Strauss said yesterday officials hoped to obtain agreement on a uniform international pricing formula for steel products that would climinate some of the advantages now given to foreign producers.

The subcommittee heard yesterday from a parade of industry and academic witnesses, who outlined the problems of the domestic steel industry, ranging from low levels of investment here at home to competition from subsidies other nation give their producers.

The only major criticism of the industry came from William D. Nodhaus, a member of the President's Council of Economic Advisers, who argued that while steel makers faced some real economic hardships some of their problems were their own making.

Nordhaus pointed out that while overseas producers had cut prices during the recession, U.S. steel makers had "steadfastly increased prices" in "an utter lack of responsiveness to market conditions. We can hardly be surprised," he said "when foreign producers undersell our own."

The presidential adviser noted that production costs also had risen sharply because of the industry's latest labor settlement, which guaranteed lifetime job security to members of the United Steel Workers' union.

"Yielding to pressures for massive import relief will not solve the problems both here and abroad," Nodhaus told the panel. "In the short run, there is only one solution to the steel problem - rapid economic growth and continuation of the recovery."

But Nordhaus' criticisms were drowned out by complaints from the subcommittee members, who made it clear that despite what seemed to be a hiatus, pressures were building anew in Congress for protectionist legislation.

Rep. Charles A. Vanik (D-Ohio), the panel's traditionally liberal chairman, repeatedly berated Strauss and a group of other administration trade officials for not acting forcefully enough to deal with the industry's problems.

"What we're dealing with is a problem that could turn into a grounds-well before we reach any solutions," Vanik warned the officials. He told Strauss to "convey to the Europeans" that unless some steps are taken "this will be stopped unilaterally by the Congress."

Other on the panel were equally vehement on the issue. Rep. Charles J. Carney (D-Ohio) announced that 60 House members were preparing to meet Thursday to form a congressional "steel Caucus" to try to push through import-quota legislation.

And Rep. Ken Holland (D-S.C.) served notice that he and other Southerners might add to that total. "Despite the cries of protectionism," Holland said, "I'm ready to join any coalition on this."

The U.S. Steel Corp. petition charging dumping named six big Japanese steel exporters as defendants - Nippon Steel Corp., Nippon Kokan, Kawasaki, Sumitomo, Kobe and Nisshin. All were charged with selling here below their domestic prices.

The document alleges the six have engaged in unfair trade practices ivolving sales of five categories of steel products - structural shapes, structural grade plates, hot and cold rolled sheet, galvanized sheet and welded standard pipe.

Under usual procedures, the Treasury would have 30 days to decide whether the petition was in order, and then would be given an additional six months to complete an initial investigation of the charges.

However, in a rare move, U.S. Steel also has charged that the Japanese firms are selling goods here at prices below the cost of production - a new ground for complaint permitted under recent revisions to the anti-dumping laws. Officials said yesterday that could lengthen the time required for investigation.

Treasury officials said U.S. Steel was listed along as the complainant. Two weeks ago, Edgar B. Speer, the company's president, had said he hoped to persuade other steel makers to join in the action U.S. Steel spokesmen had no comment on the move.

In his testimony yesterday, Strauss conceded that U.S. had been unable to persuade the Japanese to act voluntarily to limit its steel exports here. But he insisted that imposition of quotas was too severe a move. Instead, he invited American steel makers to file more anti-dumping petitions.

The congressmen's concern about steel imports apparently was heightened by the announcement on Tuesday that Youngstown Sheet and Tube Co., one of the nation's ten largest steel plants, plans to close its Campbell, Ohio, facility and eliminate 5,000 jobs.

The action was the latest in a spate of plant shutdowns by steel makers which have occurred in recent weeks. Steel producers say demand for their products now is so low and production costs so high, that they cannot afford to maintain marginal facilities.

A primary complaint of U.S. steel producers is that overseas governments have added dozens of new subsidies and tariffs to protect their own steel industries, which American steel makers say allows their competitors to undersell them in the world market.

Strauss and other officials contend much of the domestic industry's problem stems from sluggish demand in the midst of the slow recovery. Strauss predicted that excess capacity would continue for several years, but warned there "are no simple solutions."