U.S. Trade Representative Clayton Yeutter has raised the threat of an embargo on steel shipments from America's closest allies in Europe if they fail to set new limits on their exports to the United States by the end of the week.

Such a drastic step probably would touch off a cycle of retaliatory trade restrictions on both sides and sour U.S. relations with its Western European allies.

The administration is turning up the heat in the steel talks, which have been going on here and in Brussels for six weeks, because it sees a cutback in European imports as critical to the success of President Reagan's attempt to give the beleaguered steel industry time to become internationally competitive.

While other steel suppliers -- including Canada and Japan -- have cut back their shipments in response to Reagan's call for voluntary restraints 13 months ago, the European producers have increased their sales to the United States 17 percent in the first eight months of this year.

The U.S. stance is being watched closely by Congress, other domestic industries and American steel makers as a litmus test of President Reagan's aggressive new policy to reduce record trade deficits.

While Yeutter declined to specify what retaliatory action he would take against the 10-nation European Community if no agreement is reached by tomorrow, he raised the possibility of an immediate embargo because EC steel makers already have exceeded this year's quotas in some categories.

"One of the immediate issues that will have to be faced by the European steel companies is whether they are reaching the limits of their steel shipments," Yeutter said in what aides later called a deliberate reference to a possible embargo.

Sir Roy Denman, the EC ambassador in Washington, said any U.S. retaliation "would hurt" the steel talks, as well as negotiations to resolve a dispute over European restrictions on U.S. citrus fruit. The citrus quarrel degenerated briefly into a "pasta war" this summer as the United States retaliated against the citrus curbs by raising tariffs on European pasta. A truce to negotiate an end to the dispute expires tomorrow.

"Let's sweat out the next two days," said Sir Roy.

Some progress was reported yesterday in Brussels, but it appeared unlikely that an agreement would be reached by the deadline.

Yeutter acknowledged that U.S. retaliation "would change the tenor of the discussions" over steel, but added, "If this agreement is not reached by Nov. 1, we will take what action we deem to be proper, and announce it to the world." He said the United States is sticking to the Oct. 31 negotiating deadline, agreed to by both sides, even though the 1982 U.S.-EC steel pact doesn't expire until the end of the year.

"Deadlines are deadlines," Yeutter said. "I just don't like fuzzy deadlines. It's better from the negotiators' standpoint to have definite dates."

The exchange between the two officials took place over Sir Roy's dining room table Tuesday night during an on-the-record dinner he gave for Washington reporters covering trade issues.

While American steel makers play no official role in the import talks, they have been exerting pressure on the U.S. negotiators. In a September letter to Yeutter, for instance, Donald H. Trautlein, chairman of the American Iron and Steel Institute and head of Bethlehem Steel Corp., complained that Europe had "grossly and persistently violated" past restrictions on its steel shipments.

He demanded a rollback in the EC share of the U.S. market in order to achieve President Reagan's goal of limiting steel imports to 18 1/2 percent of sales here. Trautlein added that any agreement should cover all steel products, including semi-finished steel, and should be so tight, "given the past history of European violations," that there are no loopholes that would allow European producers to increase their sales.

"The time has come to make entirely proper U.S. demands and exact the requisite penalty for EC violations by very substantially reducing across the board" the European share of the American market, Trautlein said.

"Nothing less will satisfy the steel industry and, in our view, the Congress, the public and our national economic interest."