A collision last month behind closed doors between Secretary of State Cyrus Vance and a Senate subcommittee poses a SALT dilemma for President Carter that could dwarf other difficulties.

Vance's performance before the Armed Services subcommittee headed by Sen. Henry M. Jackson (D-Wash.) bordered on the disastrous. No master of the subject, the Secretary could not reassure the senators about new U.S. concessions in strategic arms limitation talks with the Soviet Union. Not only did this clarify Jackson's role as a nearly certain opponent of a SALT II treaty (in alliance with conservative Republicans), but moreover some fence-sitting Democrats joined him.

The result: SALT II is not merely "in trouble," as the White House concedes, but surely would fail ratification in the Senate today. Since nobody believes the Russians will retreat on issues tentatively decided at Geneva, it is impossible now to renegotiate a treaty more to the Senate's liking. So, the apparent alternative to defeat in the Senate is for the President to put SALT on the shelf for now.

But to do that, Carter would have to renege on his promise, made on the political barnstorming trail last month, of a SALT II agreement within weeks. Besides, his national security advisers believe reason and approval will ultimately come in the Senate. Obviously, the White House has no conception of the mood on Capitol Hill.

But Vance has. "I was amazed how [WORD ILLEGIBLE] really brutal, Scoop [Jackson] as on Vance," a Republican on the subcommittee told us. Conservative Republican senators who had feared that Jackson would bow to party loyalty and support the treaty were reassured.

Jackson was not the only Democrat to upbraid Vance. Georgia's Sen. Sam [WORD ILLEGIBLE] tongue-lashed Vance for avowed [WORD ILLEGIBLE] of an alleged U.S. offer to retance the neutron bomb. Nunn lectured the Secretary of State that he should know, even if he does not know.

Sen. John Glenn of Ohio, no hardliner in the Senate, assailed inadequate verification of Russian compliance with SALT II. Unless he can be satisfied, Glenn will be another Democrat voting against the treaty.

Only a few such Democrats are needed to reach the 34 votes required to reject the treaty in light of heavy Republican opposition - perhaps 28 out off 38 Republican senators.

The elusive Senate Republican leader, Howard Baker, has signaled which way he is going by signing up Dr. Fred Ikle, who as U.S. disarmament chief battled Secretary of State Henry Kissinger over arms control at the end of the Ford administration. In the October issue of Fortune, Ikle argues against arms control "at the price of jeopardizing the nation's independence and future" - a theme expected in a forthcoming speech by Baker.

To convince senators that concessions at Geneva do not actually endanger the nation's future, a virtuoso performance by Vance before the subcommittee was needed. It was anything but that, as seen by Vance's response to this question: Will the SALT II limit of 800 multiple-warhead intercontinental ballistic missiles ensure survival of U.S. Minuteman missiles?

Yes, said the Secretary of State, a "study" proves it. Jackson directed Vance to supply it. But at the next hearing, Vance confessed there is no "study" - only separate "assurances" about the Minuteman. All right, said Jackson, bring those in. Vance agreed, but Jackson will have a long wait; reliable Pentagon sources say no such "assurances" exist.

Meanwhile, Soviet negotiators at Geneva provide no help. Administration officials months ago confided to senators they had a loophole; the 600-kilometer limit on ground-launched an sea-launched cruise missiles would not apply if these weapons were tested from planes. Predictably, the Russians found the loophole and are demanding that the 600-kilometer limit apply to such missiles, no matter how they are tested.

If the United States accepts this demands, heartsick middle level officials at the Defense Department are poised to go public against SALT II. The President's escape hatch would be to throw up his hands at Soviet intransigence and recess negotiations.

This is precisely what would have been expected early this year when Carter was talking tough on SALT. But defense experts who had hoped for so much from him now fear Jimmy Carter is a utopian disarmer, more William Jennings Bryan than Richard B. Russell. How he handles his SALT dilemma will determine whether these fears are realized.