The Senate Foreign Relations Committee yesterday rejected another effort by Minority Leader Howard H. Baker Jr. to send the SALT treaty back to the negotiating table after a theatrical debate between Baker and presidential counsel Lloyd N. Cutler.

For the second successive day the issue before the committee was the 308 "heavy" intercontinental ballistic missiles that SALT II (like SALT I) permits the Soviet Union to maintain, though the United States is denied the right to build equally enormous weapons.

Baker yesterday offered two more amendments aimed at these missiles, after losing an 8-to-7 vote Tuesday on a proposal to allow the United States to match them.

Cutler, chief administration spokesman in the Foreign Relations Committee's markup proceedings on the treaty, then stepped out of the relatively passive role he has been playing and sharply challenged Baker's position.

Addressing the minority leader (and undeclared candidate for president) directly, Cutler said, "It may be Sen. Baker, that you will have the privilege to conduct [the] negotiations for SALT III," implicity inviting a future President Baker to see what he could to negotiate away the Soviets' heavy missiles, which are a central part of their arsenal.

"But I respectfully submit," Cutler went on, "that it would be beyond even your capability to negotiate [reduction or elimination of the 308 heavy missiles] as part of SALT II." Nothing that the joint chiefs of staff had specificially rejected reopening the negotiations on Salt ii, Cutler went on: "It would be a very heavy responsibility for anyone voting for your amendment to reject that military judgment and substitute your own. . ."

Replied Baker: "This nation is in grave peril in my judgement because you did a bad job in negotiating this treaty," allowing the retention of "these most monstrous of all weapons," the Soviet heavy missiles. Baker said he couldn't wait for SALT III to solve this problem, adding that it was a "bugaboo" argument to contend -- as Cutler did -- that SALT could be lost entirely by adopting amendments like Baker's.

In another development yesterday, informed sources said a backstage effort to produce a new consensus on a Senate declaration demanding significant reductions of strategic arsenals in any future SALT agreement had run into difficulty.

Majority Leader Robert C. Byrd (D-W.Va.) had asked Alan Cranston (D-Calif.), Daniel Patrick Moynihan (D-N.Y.), George McGovern (D-S.D.) and others to work together on this issue, but Moynihan and McGovern are so far apart at present that no progress has been made, the sources said.

According to an aide to McGovern, he is now seeking support from other senators for a proposed Senate declaration demanding future reductions that McGovern hopes to offer to the Senate Foreign Relations Committee next week.

The idea of demanding substantial reductions in SALT III appeals to senators at both ends of the political spectrum. Byrd and the White House hope that a consensus position will emerge on this issue and on some kind of declaratory commitment to increased defense spending, and that this combination will help persuade the needed 67 senators to vote for SALT II.

Yesterday's Baker amendments were both defeated by identical 9-to-6 votes. John Glenn (D-Ohio), who had voted with Baker Tuesday when he proposed granting the United States the right to match the Soviet heavy missiles, voted against him yesterday.

The first amendment Baker offered yesterday would have required the Soviets to dismantle their 308 heavy missiles by the end of 1981, and reduced accordingly the SALT II limits on ICBMs equipped with multiple warheads. The second would have required the United States and Soviet Union to negotiate an agreement by the end of 1981 to make those same reductions later, or the SALT IItreaty would lapse at that time.

Ambassador Ralph Earle, the chief U.S. SALT negotiator, said Baker's preoccupation with the heavy missiles was misplaced, adding that "if every Soviet Ss18 [heavy missile] sank into the sea tomorrow," remaining Soviet weapons could pose what amounts to a comparable threat to the United States.

Frank Church (D-Idaho", chairman of the committee, indirectly accused Baker of playing politics with his amendments. Church said of one Baker proposal, "This is an amendment without a military purpose."