President Reagan was warned yesterday that congressional approval of an $8.5 billion increase in the U.S. contribution to the International Monetary Fund will be jeopardized if he fails to impose stiff penalties on imports of specialty steel.

The president faces a Tuesday deadline in a knotty specialty steel unfair-trade-practices case that reportedly has split the Cabinet and White House staff despite recommendations from the International Trade Commission that three-year quotas on imports be imposed. Sen. John Heinz (R-Pa.) said six options developed by the White House staff were flown to California yesterday for the president's decision.

"I don't know what the president is going to decide. I hope he picks one of the strong ones," Heinz told a special hearing run by the other senator from Pennsylvania, Republican Arlen Specter, designed to highlight problems in the American steel industry.

Heinz said steel caucus congressmen are likely to oppose increased IMF funding if they are disappointed by the specialty steel outcome because of constituent complaints that the U.S. contributions help subsidize the Brazilian steel industry, whose exports to the United States are blamed for the loss of American jobs. Heinz said steel caucus votes are need for House passage of the IMF legislation.

Heinz's warning was given unusual credence on Capitol Hill because he fought hard for Senate passage of the IMF contribution.

The increased IMF funding, needed to help Third World nations such as Brazil out of their debt crisis, has come under attack as a bail-out for big bankers. House members reportedly are being flooded with mail urging them to oppose the IMF funding.

But this new connection between specialty steel trade sanctions and the IMF funding adds another wrinkle to the already complex international banking crisis and provides another illustration of the interdependence of the economies of the world. It also shows how vulnerable the American steel industry--once the most powerful in the world--is to the global economy.

United States Steel Corp. Chairman David M. Roderick said the industry lost $3.2 billion last year. He predicted losses will continue until early 1984 despite what he called a "slow recovery." Steel economist William T. Hogan of Fordham University said the industry shut down 10 million tons of steel-making capacity last year and is likely to drop another four million to five million tons during the next 18 months.

Roderick, whose company remains America's largest steel producer, called for "modest" and "temporary" limits on steel imports to give the U.S. industry time to adjust to the new realities of the global industry and to protect it from "devastating" subsidized imports.

He also proposed changing America's antitrust laws to recognize the global nature of the steel industry by defining it in terms of the world, not just this country. This would make it easier to "rationalize" the industry through mergers "which create more efficient steel companies," he said.

But Chairman Dennis J. Carney of Wheeling-Pittsburgh Steel Corp., one of the smallest of the industry's top 10, opposed easing antitrust laws. "Rigorous enforcement of antitrust laws is essential to protect consumers from adverse effects of greater concentration of economic power in our industry," he said.

Roderick also defended U.S. Steel's proposal to bring in steel from Britain to be finished in the Fairless Works outside Philadelphia as necessary to preserve jobs there.

But United Steelworkers of America official James N. McGeehan said the plan means the loss of 3,000 jobs because of the closing of Fairless' hot metal operations.