The Senate Foreign Relations Committee yesterday failed to muster majorities for either a nuclear freeze resolution or an alternative calling for an arms "build-down," and finally decided to send both to the Senate floor with notations that it agreed with neither.

The Democratic-controlled House adopted a freeze resolution earlier this year. The votes yesterday indicated the Republican-controlled Senate may not be able to rally behind any arms-control position.

The Reagan administration has resisted the freeze idea as simplistic.

The freeze resolution, sponsored by Sen. Mark O. Hatfield (R-Ore.) and Sen. Edward M. Kennedy (D-Mass.), was defeated 10 to 7 as Sen. Edward Zorinsky (D-Neb.) joined the committee's nine Republicans in opposition.

The surprise of the afternoon then came when a substitute crafted by Chairman Charles H. Percy (R-Ill.) also failed on an 8-to-8 tie as Sen. Charles McC. Mathias Jr. (R-Md.) joined the panel's seven Democrats in voting no. Zorinsky did not vote.

The Percy measure, which was co-sponsored by Sen. Larry Pressler (R-S.D.) and Sen. Rudy Boschwitz (R-Minn.), was considered the most likely rallying point for those who oppose a freeze but want to press the president to move faster on arms control.

Under the build-down proposal, the United States and Soviet Union could continue to deploy new weapons, which a freeze would forbid. But for each new weapon deployed more than one old weapon would have to be retired.

Neither the freeze resolution passed by the House nor the arms resolutions sent to the Senate floor yesterday would be binding. But they are important symbolically, and the arms issue is likely to be an important one in next year's elections.

The freeze movement is national and well-organized. A third of the Senate, including such highly visible Foreign Relations members as Percy, is up for reelection. In addition several Democratic senators, including pro-freeze Foreign Relations member Alan Cranston (D-Calif.), are candidates for president.

The committee at some points yesterday seemed to be wrangling as much over parliamentary phraseology as nuclear policy.

Cranston at one juncture proposed that both defeated resolutions be sent to the floor "without prejudice." Percy objected; he wanted to send the freeze plan to the floor with a negative recommendation and his without any. The committee finally agreed to report both "in disagreement."

The build-down proposal, put forward early this year by Sen. William S. Cohen (R-Maine) and Sen. Sam Nunn (D-Ga.), has drawn the support of 43 other senators.

In the spring President Reagan also announced his support for the idea. He was then seeking congressional approval for the MX missile. Congressional moderates said that they would support the missile only if Reagan would agree to new steps toward arms control, of which build-down was one.

Despite Reagan's endorsement of the idea, his arms-control specialists have been unable to agree on how to include it in the U.S. strategic arms negotiating position.

Pressler had urged the administration to include such a proposal in its negotiating position as an alternative to a freeze. He said yesterday that he is disappointed that no such proposal has been forthcoming. He also said rather pointedly that "there will be one before the next MX vote."

During the afternoon, the committee approved a proposal by Sen. Jesse Helms (R-N.C.) that the administration seek compliance with the threshold test ban treaty.

That treaty, signed in 1974 by President Nixon, calls for both the United States and the Soviet Union to limit their underground tests to 150 kilotons or less. Though the treaty has not been ratified, both countries have agreed to abide by its provisions.

Before the vote, Helms displayed a Defense Department chart, which he said had been declassified, showing that 15 Soviet underground tests since 1978 have exceeded the limit and that four were almost double it.

Committee member John Glenn (D-Ohio), also a Democratic presidential candidate, said during yesterday's debate that in future arms-control proposals the United States must begin to count the nuclear weapons of such allies as England and France, which the Reagan administration has resisted doing.

"It is inconceivable," Glenn said, that the Soviet Union would undertake substantial reductions of its nuclear stockpile "unless it gets some kind of agreement with other states."