The Senate Foreign Relations Committee voted yesterday to let the controversial nomination of Kenneth L. Adelman as director of the Arms Control and Disarmament Agency go to the floor but with a recommendation that the Senate defeat it.

The vote giving an unfavorable recommendation to the nomination was a stiff rebuff to President Reagan on arms control, but it was less than a total loss for him. The alternative was to bottle up the nomination in committee.

Senate Majority Leader Howard H. Baker Jr. (R-Tenn.) said after the vote that he still thought Adelman would be confirmed.

But the floor vote is likely to be delayed, the nomination is sure to become the vehicle for an extended debate on arms control, and Baker acknowledged, "it's not going to be easy."

Adelman was chosen by Reagan to succeed Eugene V. Rostow, who was fired in the midst of controversy over administration arms control policy earlier this year. Despite three appearances before the Republican-run committee, Adelman failed to win a majority of members over to his side.

Opponents have said that he has neither the experience nor the commitment to arms control necessary for the job.

Sen. Charles McC. Mathias Jr. (R-Md.), one of the two committee Republicans who joined seven Democrats in opposing Adelman, said after the vote that it may be weeks before the nomination gets to the floor; a Baker aide confirmed that it would be "mid-March."

Sen. Paul E. Tsongas (D-Mass.) said yesterday that he will filibuster when the nomination comes before the Senate.

The 17-member committee yesterday first defeated, 9 to 8, a motion by Chairman Charles H. Percy (R-Ill.) to report the nomination favorably. One Democrat, Edward Zorinsky (Neb.), joined the seven Republicans in favor.

Baker then moved to send the nomination to the floor "unfavorably" as a way of salvaging it.

The Baker motion passed 14 to 3. Had the nomination been bottled up in committee, the administration would have had to persuade the Senate to approve a discharge petition to force it to the floor, a difficult procedural task under the best of circumstances.

In a caucus before the vote, Baker claimed he eventually could find the votes to discharge the nomination from the committee, but said that the process would hurt the prestige of the committee and the Senate.

"We did not have the same oomph for bottling Adelman up in committee," one senator said yesterday, "and since it was only a one-vote margin against him, we decided to give it up."

Mathias said the committee's action represented a tradeoff, in that the discharge petition would have been brought up immediately, not giving opponents time to put together a committee report and prepare for a debate.

Now, he said, there will be time for extensive preparation for a debate on Reagan's arms control policies generally as well as specifically on the qualifications of Adelman, who is the deputy U.S. representative to the United Nations.

Since 1925, Senate committees have sent only 11 nominations to the floor with unfavorable recommendations. Only two of the 11 negative reports were overturned and the nominees approved.

Earlier in his administration Reagan nominated Ernest W. Lefever as assistant secretary of state for human rights and humanitarian affairs.

The Foreign Relations Committee sent that nomination to the floor with a negative recommendation. It was delayed there, and Lefever subsequently withdrew.

During the four-hour hearing that preceded yesterday's vote, Tsongas asked Adelman if he had considered withdrawing, particularly after the committee voted 15 to 2 last Wednesday to delay acting on his nomination in hopes the president would reconsider it.

Adelman said he had not, that he "learned quickly that the president has full confidence in me, I cherish that very much."

Tsongas asked if he thought that "in persisting" in the nomination Adelman was serving the best interests of the president? "I do," Adelman replied.

"We're just going to let Adelman wait a while," one senator opposed to the nomination said yesterday.

Meanwhile, he added, some members "might get sensitized by the West German elections and the major nuclear freeze lobbying effort" scheduled for the first week in March. Arms limitation is expected to be a major issue in the West German elections.

The first portion of yesterday's committee session was devoted to questioning Adelman and New York Daily News columnist Ken Auletta, who in 1981 quoted Adelman as having said arms negotiations were a "sham."

Adelman in a statement last week said he could not recall having given Auletta an interview and denied having called arms negotiations a sham. He also said that had not ever been his view of such talks.

Yesterday Auletta produced six pages of notes and a telephone bill that listed a 25-minute call to the Stanford Research Institute in Arlington, Va., where Adelman worked at the time.

There was one light moment in the questioning. Adelman supporter Sen. Jesse Helms (R-N.C.) noted from Auletta's telephone bill that after he supposedly called Adelman he had placed a call to the Capitol switchboard. Which senator had he called, Helms wanted to know.

It was not until after Helms left the hearing room that Auletta discovered from his notes that he had called John Carbaugh, an aide to Helms, a disclosure that brought laughter from the hearing room.