THE UNITED STATES is in something of a box in protesting against the Israeli cabinet's troublesome and wrongheaded decision to set up a new West Bank settlement, the first announced since the peace treaty with Egypt. If the administration protests too little, it invites the accusation that it privately winks at new settlements. If it protests too much, it risks confirming the accusation that Israel has no intention of going beyond a Sinai deal: hence others would not be wise to join Camp David. So it was that the State Department termed the cabinet decision "harmful" to the peace process and "regrettable" in coming just as Egyptian-Israeli talks on Palestinian autonomy opened, but indicated it would not press the matter further. It wants those talks to move.

The decision on the new settlement barely got a majority. Eight ministers, including Menachem Begin, supported it. Seven, including the heavy-weights at defense, foreign affairs and finance, voted against it or abstained, mostly on grounds that it would embarrass Egypt and Israel alike at a delicate time. This is a very narrow margin on which to conduct so politically costly a policy.

It furnishes some reason to expect that Israelis will not engage further in provocative declarations of a right to create new settlements, and that they will keep new settlements few and small. Certainly a government contending that settlements will not disrupt negotiations will want to prove so - by doing its part to make those negotiations work. Such a government should alos be eager to prove it has not either slyly or feebly surrendered its authority on the West Bank to Israeli thugs, those religious fanatics who abuse - and sometimes even kill - Arab residents. They are no different from the Palestinian terrorists who kill Jews.

The prevailing theory when the peace treaty was signed in March held that the other Arabs had to be brought along quickly to make the treaty stick. Frustrated in that effort, the United States, or at least Jimmy Carter personally, has fallen back on an alternative theory. It holds that a demonstration that peace is permanent and works to the advantage of those who support it is the best way to proceed. Egypt and Israel have been doing well in this regard in their relations with each other. In their talks on the Palestinian question, they have hardly done more than state opening positions.

Mr. Carter's contribution has been to underline a moderate position appreciated in both Cairo and Jerusalem - that an independent Palestinian state would be "destabilzing" - and to indicate that the United States will not "preempt" Egyptian-Israeli negotiations by putting forth its own ideas prematurely. It is not a policy assured of success, but it needs and deserves some time to show what results it can achieve.