THE new Marine casualties in Lebanon may strain still further the public's confidence in administration policy in Lebanon. There is in the United States a special warmth for this small, much-abused, Western-oriented place, but there is not an unlimited readiness to see American lives lost in the obscure, grinding conflict there. The need grows for Mr. Reagan to make clear what diplomatic strategy the Marine presence is intended to serve.

Americans tend to forget that Israel's invasion merely interrupted a long and painful civil war. With the Israeli decision to pull back to the Awali River, a new chapter in it opened. Which Lebanese would take power in the evacuated area? The Druze, traditionally dominant in the Chouf mountain region, feared the Israelis were handing it off to a Christian Phalange militia working hand in glove with the Lebanese army. The Shia Moslem militia expressed similar fears of what would follow Israeli withdrawal from suburbs of East Beirut. Both Druze and Shia saw the Marines as patrons of the Christian establishment. That's a considerable part of the reason why the latest shooting began.

For years the United States has described its policy in Lebanon as trying to counter community splits and external intervention by building a strong central Lebanese government with a reliable national army at its command. Like its predecessors, however, the Gemayel government is seen by major social groups as tipped to favor an upper-class and/or Christian minority. The more that government is built up, the greater the distrust expressed --with guns as well as words--by other groups.

American diplomats have spurred on the tentative steps President Gemayel has taken to open what he calls a dialogue of national reconciliation. But more must be done. Mr. Gemayel's government is suspended between its national duties and its Phalangist roots. Revising the Lebanese system to reflect the demographic and political changes since the country's basic power-sharing terms were laid down 40 years ago would be a tall order if the country were unoccupied, prosperous and at peace. It is, of course, none of these things.

With their separate agendas, Israel and Syria may yet frustrate efforts by national-minded Lebanese to restore and reform their system. But certainly it is too early for the United States to conclude that no progress can be made and to pull out the Marines. Lebanon would be completely shredded. The United States would be seen throughout the area as turning tail. Congress has reason to be uneasy: its agitation over application of the War Powers Act is the evidence. The administration could help by making clear it knows where it's trying to go.