Congress and President Reagan will go head-to-head next week over a difficult question that Congresses and presidents have been debating since the birth of the Republic: who has the power to send U.S. troops into hostile situations overseas?

Ever since the sharp debate in the 7th Congress that followed President Jefferson's dispatch of Marines to the shores of Tripoli in 1801, the legislative and executive branches have sought ways to reconcile a president's need for fast and flexible response with Congress' constitutional power to declare, or not declare, war.

The old debate is expected to flame anew next week when Congress returns from recess to consider the future of the U.S. peace-keeping force in Lebanon, which has suffered four fatalities so far and is firing back protectively with bigger and bigger weapons.

Legally, the question will turn on proper interpretation of the War Powers Resolution, a 1973 law passed over bitter opposition from, and a veto by, President Nixon. The resolution was designed to prevent recurrence of extended conflicts--such as the wars in Korea and Vietnam--carried out without a congressional declaration of war.

The resolution requires a president to "consult" with Congress whenever possible before sending U.S. troops into foreign waters or territory, except for routine training or replacement. It requires the president to file reports in writing within 48 hours of the introduction of American forces overseas.

Most important, the act also sets firm time limits for such deployments--time limits that expired long ago in the case of the Marine force in Lebanon. The key question at issue now is whether those time limits should apply to the Marines.

The War Powers legislation contemplates three general circumstances in which American troops might go abroad. The first is a routine training mission or replacement of troops already on station; the law has no application to such activities.

A second situation involves U.S. troops "equipped for combat" going to a foreign outpost where no hostilities are likely. In that case, the law requires only that the president report to Congress--first when the troops are sent in, then every six months as long as they stay.

The third situation deals with the dispatch of troops "into hostilities or into situations where imminent involvement in hostilities is clearly indicated by the circumstances."

In that case, the troops must come home within 60 days--90 days in cases of "unavoidable military necessity"--unless Congress declares war or gives specific authorization for a longer deployment.

The central legal question on the Lebanon peace-keeping force is whether the U.S. Marines are in an "equipped for combat" situation, with no time limit on their stay, or involved in "hostilities" and thus subject to the 60-day limit.

When Reagan sent the Marines to Lebanon last Sept. 29 he told Congress that there was "no intention or expectation that armed forces will become involved in hostilities." The White House position has been that the "equipped for combat" provision applies, and that the president needs no congressional authorization to keep U.S. soldiers in that country.

After the first two U.S. Marines were killed by shellfire in Beirut, White House spokesman Larry Speakes said that the U.S. force is not in combat and that the time limits in the law do not apply.

But members of Congress in both houses and both parties have said that hostile fire on the Marines means that U.S. troops are engaged in hostilities. They say the president should notify Congress formally that this is so--a notification that would trigger the 60-day limit.

Congressional leaders also say that if Reagan declines to issue a formal report of "hostilities" Congress can declare that he is required to issue such a notice. That way, they say, Congress would start the 60-day clock ticking. At the end of that period, there could be a constitutional standoff in which the president keeps the Marines on station and members of Congress call the action illegal.

It is hard to say how a dispute like that would be resolved, because the situation has not come up before under the war powers legislation.

Since the resolution passed in 1973 presidents have issued eight reports. Three involved the U.S. evacuation from Vietnam in 1975. Two involved attempted rescues--of the Mayaguez crew in 1975 and of the Iran hostages in 1980. The three latest dealt with the Mideast: the Sinai Multinational Force in 1982 and two deployments of Marines to Lebanon in the same year.

None of those deployments lasted more than 60 days.