The Senate yesterday rejected, 78 to 17, a move by Sen. Jesse Helms (R-N.C.) to bar President Reagan from following through on his plan to dismantle a Poseidon nuclear submarine in order to continue compliance with the unratified SALT II treaty.

The proposal, which Helms sought to add to a $13.5 billion omnibus spending bill, would have prevented the estimated expenditure of $21 million to dismantle the missile-carrying sub until the president certified that the Soviet Union was abiding by the strategic arms limitation treaty.

Helms' move followed conservative Republicans' complaints that Reagan should abrogate the 1979 treaty, on grounds of Soviet violations, rather than continue compliance in hopes of going "the extra mile" for arms control, as Reagan announced he would do last week.

In an implied repudiation of the president's decision, Helms argued that a vote against his "Save the Poseidon" proposal was "a vote for appeasing the Soviets in the face of their very clear SALT violations."

In arguing against the proposal, Foreign Relations Committee Chairman Richard G. Lugar (R-Ind.) used uncharacteristically harsh language, describing it as "mischievous" and fraught with potentially "disastrous" implications for arms control.

The Republican-controlled Senate should give Reagan's plan a chance rather than try to "put ourselves in the role of the president," Lugar said.

Rejection of Helms' proposal followed a Senate vote this month urging Reagan to continue voluntary compliance with SALT II so long as the Soviets abide by it. That vote was viewed as a factor in influencing Reagan to continue voluntary compliance.

Voting with Helms yesterday were conservative Republicans and two Democrats, Ernest F. Hollings (S.C.) and Edward Zorinsky (Neb.).

Helms' move against SALT II marked the second time in as many days that he crossed swords with the Senate Republican leadership on an issue involving a compromise with the administration.

Earlier, Majority Leader Robert J. Dole (R-Kan.) sidelined a bill incorporating a compromise with the White House on small-business programs after Helms threatened to try to attach a proposal that would sanction prayer in public schools and would have prompted a filibuster from foes of the prayer provision.

The small-business measure was sponsored by Sen. Lowell P. Weicker Jr. (R-Conn.), who led the successful fight last year against a Helms-backed constitutional amendment to allow prayer in public schools.

Meanwhile, in back-room negotiations on the $13.5 billion omnibus spending bill, Senate leaders and White House officials cleared one major hurdle, negotiating an agreement on cost-sharing for water projects that had been threatened with a presidential veto.

They agreed to $63 million in initial funds for 25 flood-control, harbor-improvement and inland-waterway projects so long as cost-sharing agreements could be worked out within a year with local governments and affected commercial interests. But they still lack agreement on long-term provisions for cost-sharing on all such projects before the veto threat will be withdrawn, budget director David A. Stockman said after meeting with Senate leaders.