The latest hostilities in the Persian Gulf prompted new demands on Capitol Hill yesterday for constraints on U.S. tanker-escort operations, as well as suggestions of compromise to avoid a confrontation between the White House and Congress over invoking the 1973 War Powers Resolution.

One proposal under study by Senate leaders last night would require congressional approval for a continuation of the escort operation after a specified period of time, but under procedures separate from the war-powers law, which the administration has vowed not to invoke.

While most lawmakers supported the U.S. helicopter attack Monday on an Iranian naval vessel that was reported to be laying mines in the gulf, many expressed fears of a widening, open-ended U.S. involvement in the Iran-Iraq war, with constant risk to American lives.

Both houses of Congress have repeatedly expressed reservations about U.S. escorts in the gulf of Kuwaiti oil tankers that now fly the U.S. flag, but they have stopped short of acting to invoke the Vietnam-era war-powers law, which requires congressional approval for sustained U.S. combat operations.

Last Friday, the Senate voted 50 to 41 against invoking the law as part of the defense authorization bill for fiscal 1988. Defense leaders such as Armed Services Committee Chairman Sam Nunn (D-Ga.) argued that the gulf war did not meet the law's requirement for congressional involvement when U.S. forces are introduced into "situations where imminent involvement in hostilies are clearly indicated."

But Monday's confrontation in the gulf prompted critics of Reagan's policies to resume their push for congressional action to invoke the law. And Nunn suggested that the White House should work with congressional leaders to find a way to comply with the "spirit" if not the letter of the law.

Nunn said he is searching for a way to require the kind of consultation between the White House and Congress that the law requires without triggering a "very disruptive kind of confrontational debate" while U.S. forces at risk abroad.

"You have to take a fresh look at the situation now because, obviously, there have been some hostilities," Nunn told reporters. "It looks to me that it's time to sit down and talk about what's happening over there," he added.

Meanwhile, Sen. Lowell P. Weicker Jr. (R-Conn.) introduced a new proposal to invoke the War Powers Resolution as part of the defense authorization bill, prompting a countermove by Senate Majority Leader Robert C. Byrd (D-W.Va.) to come up with a compromise that could win bipartisan support.

One version under consideration last night would curtail funding for the escort operation after a specified period of time unless Reagan certifies to Congress that a continuation is needed. At this point, Congress could act under expedited procedures to disapprove the certification by a majority of votes in both houses.

It is expected that a Senate proposal along these lines will be acted upon today, leadership sources said.

Aside from compromise efforts by Byrd, Nunn and others, it appeared that Monday's hostilities had reinforced earlier positions rather than changing minds on the war-powers issue.

"The War Powers Act should be invoked, and I hope and pray that it will be because there is going to be one incident like this after another," said Sen. Brock Adams (D-Wash.), who sponsored the earlier war-powers move along with Sens. Dale Bumpers (D-Ark.), Mark O. Hatfield (R-Ore.) and Frank H. Murkowski (R-Alaska). The Persian Gulf is shaping up as "Vietnam all over again," said Adams.

"You've got people being killed, guns being shot, and {Iranian parliamentary speaker Hojatoleslam Hashemi} Rafsanjani saying the U.S. will be sorry," said Sen. J. Bennett Johnston (D-La.), who supported the war-powers move. "If that's not a warning of danger, I don't know what is."

But Sen. Dan Quayle (R-Ind.), who helped lead the fight against the war-powers provisions last Friday, contended that the U.S. attack on the Iranian ship was a defensive act against mines laid to strike tankers, not U.S. naval ships, thus falling short of provisions that would trigger the law. "If we allow the placing of mines to be an 'imminent hostility,' what's to prevent anyone from laying mines anyway and triggering the law?" he asked.

Sen. George J. Mitchell (D-Maine) also said he thinks that the war-powers provisions do not apply because they were aimed at undeclared wars and the gulf hostilities have not yet reached that point.

There were also signs of renewed pressure for some kind of action in the House, along with continued reluctance to take action that might backfire on Congress by either endorsing the policy or countermanding it with legislation.

"I think there is obviously concern, as we are involved in incidents of this kind," said House Majority Leader Thomas S. Foley (D-Wash.). "It makes it more and more difficult to avoid terms of the {war powers} act."