The Carter administration, struggling to appease European allies as well as domestic steelworkers, in preparing a modified steel trigger-price mechanism that would last a maximum of five years to give U.S. and European steelmakers a chance to retool and modernize, administration sources said yesterday.

After the five-year period the trigger-price mechanism will be abandoned and the U.S. government will begin vigorous enforcement of antidumping laws, the sources said.

Once the trigger-price mechanism is agreed upon, U.S. Steel Corp. is expected to withdraw its dumping complaints filed last March against steelmakers in seven European countries.The complaints created tensions between the two trading partners that threatened to lead to a trade war. The five years would serve as a cooling-off period and head off a possible trade war, sources said.

"The overall thrust is an attempt to adjust both at home and abroad," one source said.

If U.S. Steel wins the suits and the Europeans are forced to pay dumping duties, they may withdraw from U.S. markets, making it easier for the entry of legitimately cheaper steel from newer, more efficient plants in places such as Canada, Mexico and Brazil. Then, more expensive, domestic steel would suffer, sources said.

The U.S. Steel complaints were filed after the administration refused to raise the trigger prices during the second quarter this year. In return, administration officials suspended the trigger prices which are intended to start an investigation automatically of import activity when goods are sold here at prices under specified levels.

The steel industry has been dissatisfied with the trigger prices, claiming that they are too low and place domestic producers at a competitive disadvantage. However, industry officials have said they need a modified trigger-price mechanism to prevent dumping.

In addition to placing a five-year limit on the trigger-price mechanism, the administration also plans to raise trigger-price levels about 10 percent to reflect increased costs of production and create an antisurge mechanism that would prevent an influx of low-priced steel during the five-year period, sources said.

Administration officials now are seeking assurances from the EEC that the five years would be used by the Europeans to become more efficient steelproducers so that dumping wouldn't be necessary, sources said. In the past, the Europeans have been urged by the EEC to modernize their plants voluntarily, but they have resisted those efforts because it would be too expensive. The administration hopes that the Common Market council of ministers would impose modernization programs on their steel industries.

U.S. Trade Representative Reubin Askew has been meeting with the Europeans for a week on steel and other issues. He was quoted by an EEC official this week as saying the trigger-price mechanism will be instituted and that U.S. Steel simultaneously will withdraw its complaints. Askew, however, didn't say when the trigger prices would begin.

Speculation has been floating for the past several weeks that the trigger prices will be implemented soon. A number of technical points remain unresolved, sources said.

Some domestic steel industry officials had wanted a two-tiered trigger-price mechanism, one for the Europeans and a lower one for the Japanese who are considered more efficient than the EEC steelmakers. That idea got nowhere in the administration, sources said.

Also rejected was a suggestion that the trigger-price mechanism somehow would limit the amount of imported steel solid here.

If the Europeans thought modernization was painful before, it may be even harder now because of the world-wide recession that is lessening the demand for steel. The EEC yesterday announced that European steel output last month, compared with July, dorpped 5.9 percent. Etienne D'Avignon, EEC commission member for industry, said that he will meet next week with the EEC's 12 top steel groups to try to avoid ordering mandatory output quotas.

President Carter is expected to deliver a steel policy report soon on how his administration plans to help the ailing steel ndustry. The report, in cluding a final trigger-price mechanism, is in part a response to Republican presidential nominee Ronald Reagan who has pointed to the steel industry's plant closings and layoffs as an indictment of administration economic policies.