The Iranian missile attack on a U.S.-flagged tanker inside Kuwaiti territorial waters on Friday is forcing the Reagan administration to define with new precision its role and objectives in the Persian Gulf in what Secretary of State George P. Shultz yesterday called "the red line" demarking America's commitment.

Any decision to strike back against the presumed launch site of the Silkworm missile that hit the Sea Isle City would be likely to change radically the American posture in the gulf.

It would end any U.S. pretense of strict neutrality in the seven-year Iran-Iraq war, tilt the United States clearly to the side of Iraq, cross a new line in defense of the Arab gulf states and open the door to escalation of hostilities between the United States and Iran that could be difficult to control.

Any perceived loss of neutrality in turn is likely to undermine the U.S. position at the United Nations, where it portrays itself as an honest broker in the search for an end to the conflict.

Domestically, a U.S. military attack would be bound to sharpen the administration's troubles with Congress over the War Powers Resolution.

So far, the Democratic-led Congress has failed to invoke the resolution but, if applied, it could give Congress a potentially decisive role in gulf policy-making.

At the same time, U.S. policy-makers face other dangers if they do not take any armed action. A U.S. decision not to respond would be likely to raise questions here and in the gulf about the seriousness of the U.S. commitment to defend U.S.-flagged ships there, about the administration's readiness to run risks to prevent Iran from "bullying" the weak Arab gulf states and about the credibility of American power.

This situation is far different from appearances in mid-July, when the U.S. naval escort of U.S.-flagged Kuwaiti tankers began. Then, the administration seemed ready to strike back at any Iranian provocation.

The Pentagon let it be known that a military response could come if Iran took even the first step toward firing at U.S.-escorted tanker convoys in the Strait of Hormuz by "locking on" the passing ships with Silkworm radars.

Now, Iran has not only "locked on" to a U.S.-flagged ship, but hit it with a 1,000-pound missile warhead, seriously injuring the American captain.

An administration assumption that the mere buildup of U.S. forces in the gulf would be sufficient to deter Iran appears now to be faulty, just as a similar notion proved wrong in Lebanon four years ago. There, a huge naval buildup off the coast failed to stop attacks by Iranian-supported Shiite extremists on U.S. Marines serving in a multinational Western "peacekeeping force" in Beirut. After 241 servicemen were killed in the 1983 barracks bombing, the United States withdrew.

The lack of an American response would seem certain to embolden Iran to continue "gray area" attacks where the administration has yet to clarify its "rules of engagement," such as the situation with the Sea Isle City inside Kuwaiti territorial waters.

Yesterday, President Reagan in his weekly radio message vowed to continue protecting American-flagged ships in the gulf and said any attempt to interfere with them would be dealt with "appropriately." Freedom of U.S. navigation in international waters, he said, is "a cardinal principle" of U.S. foreign policy.

But as if underscoring the quandary Washington faces, he provided no hint of what he intends to do in response to the Iranian missile attack on the Sea Isle City.

The presumed location of the Silkworm launch site is particularly significant. It is at the extreme southern tip of Iraq's Faw Peninsula, seized in 1986 and still held by Iran.

A U.S. strike there would aid Iraq militarily and protect Kuwait from further Iranian attacks, two roles the administration has so far carefully avoided playing in the conflict.

Unlike the five other Arab gulf states that are partners of Kuwait in the Gulf Cooperation Council, the United States has no formal commitment to defend Kuwait and has refused to provide arms to Iraq or even train its military.

Now, however, Arab gulf rulers and some key congressional leaders are pressing Washington to pass through these thresholds and become involved in what Rep. Les Aspin (D-Wis.), chairman of the House Armed Services Committee, calls a "policy of containment" of Iran's "revolutionary messianism."

This would alter the nature of the Iran-Iraq war toward what gulf military specialist Anthony H. Cordesman calls "a fundamental duel" between Iran and the United States. "Neither side can easily afford to fail," he said in an interview.

Cordesman noted that grave dangers face the administration if it accepts the Iranian challenge. "Handling the escalation ladder is one thing no power has done well historically," he said.

Administration policy-makers, acutely aware of miscalculations in the past three months about Iran's readiness to risk direct confrontation with the United States, seem to be weighing closely this time how -- or whether -- to make the next step up that ladder.

From the beginning, administration policy-makers offered changing definitions of U.S. objectives, ranging from the limited task of protecting only the 11 reflagged Kuwaiti ships to defending the free right of navigation in the gulf's international waters for all shipping.

In the Pentagon's first formal outline to Congress of the U.S. gulf mission, Defense Secretary Caspar W. Weinberger on June 15 promised that U.S. protection of the 11 Kuwaiti tankers was "not part of an open-ended unilateral American commitment to defend all nonbelligerent shipping in the Persian Gulf." In the text, the word "not" was underlined.

However, on Oct. 9 after the U.S. helicopter attack on four Iranian speedboats, Weinberger offered a much broader view of the U.S. gulf role.

"We're there to ensure that there will be the free passage of vitally important cargoes in international waters," he said. As part of "the normal course of patrolling" those waters, U.S. military forces would "make sure of no additional mine laying, no additional concentrations that might be attempting to interfere with the free passage of navigation."

A few days later, Weinberger took sharp issue with congressional charges that administration objectives were "vague" and asserted that the U.S. mission in the gulf was indeed "open-ended."

"Our military objectives are not only not vague," he said, "they are quite specific and vital to the national security. They are simply to pursue freedom of the seas and the freedom to pass freely over international waters."

This commitment, he added, is "of course opened-ended because this is indeed a cardinal and permanent policy of the U.S. government."

Similar variations in stating U.S. gulf objectives have emerged in the administration's current internal debate about how to respond to the Sea Isle City attack.

The "rules of engagement" laid down for U.S. warships escorting reflagged Kuwaiti tankers do not now extend into Kuwait's waters. But several other stated U.S. commitments would seem to require some U.S. action.

For example, State Department spokesman Charles E. Redman at a Sept. 9 news briefing noted that while there was no certainty what the United States might do if a U.S.-flagged ship was hit by Iran inside Kuwait's waters, "we protect U.S-flagged shipping throughout the world, wherever it is."

The other commitment, renewed by Shultz as recently as Thursday, has been to tell Iran the United States is determined to show its resolve to help its Arab gulf friends defend themselves against Tehran's threats and keep the Soviet Union out of the gulf.

The U.S. purpose, he said, is "to see to it that Iran did not succeed in becoming dominant in the Persian Gulf by intimidating and bullying the {Arab} gulf states, and that the Soviet Union did not become, in a sense, the protector of those vital supply routes."

The Iranian missile attack is testing in dangerous new ways whether, and how, the administration intends to uphold these stated commitments and objectives in the gulf.