President Reagan suffered a series of setbacks on arms policy yesterday, as his plan to end U.S. compliance with the SALT II strategic arms treaty came under assault on both sides of Capitol Hill and committees in both houses moved to slash his "Star Wars" missile defense system.

The Democratic-controlled House, on a vote of 256 to 145, approved a nonbinding resolution calling on Reagan to continue adherence to the weapons limits set in SALT II. Thirty-seven Republicans defied the administration to vote with the Democratic majority.

At the same time, the Republican-dominated Senate Armed Services Committee unexpectedly attached a similar nonbinding SALT II amendment to the fiscal 1987 Defense Department authorization bill. Sponsored by Sen. Gary Hart (D-Colo.), the amendment was approved, 10 to 9, with Sen. William S. Cohen (R-Maine) joining the Democrats.

On the president's Strategic Defense Initiative (SDI), a House Armed Services subcommittee voted yesterday to cut $1.7 billion out of the administration's request for $5.4 billion for fiscal 1987, while the Senate Armed Services Committee appeared ready to approve a similar reduction today.

The Senate committee is expected to adopt an amendment by Sen. Sam Nunn (D-Ga.), the panel's top-ranking Democrat, that not only would cut the president's request by $1.7 billion, but would demand that it be channeled toward research that would not violate the 1972 Antiballistic Missile Treaty.

Yesterday's votes on the SALT arms control treaty, although of no immediate practical effect, represented the newest round in what has become a heated debate over arms control since the Reagan announcement last month that the United States would no longer be bound by the limits set in the unratified SALT II treaty and probably would exceed those limits later this year when an additional B52 bomber is equipped with air-launched cruise missiles.

Although the president's statement suggested a willingness to continue technical compliance with the treaty depending on Soviet behavior, other administration officials declared that the treaty is dead.

Adoption of the House resolution yesterday was seen by Democratic leaders as a prelude to a later effort to force continued U.S. compliance with SALT II. The Democrats plan to amend the House version of the Defense Department authorization bill to prohibit spending for weapons deployment that would exceed the SALT II limits.

"We hope it is not necessary to come up with binding legislation," Majority Leader James C. Wright Jr. (D-Tex.) said. "But the president must be made to understand the awesome gravity of the situation."

A Democratic aide said yesterday's vote indicated there will be a major battle over arms control this summer, but he said Democratic leaders had no hope of mustering enough votes to override a veto of binding SALT II legislation.

House Republicans assailed the resolution, accusing Democrats of blindly ignoring evidence of Soviet violations of the treaty, which Reagan cited in announcing the end of U.S. compliance.

Calling SALT II "a paper pussycat," Rep. Henry J. Hyde (R-Ill.) said assertions that the treaty had restrained the Soviets' arms buildup demonstrated that "the capacity of the left in this chamber for self-deception is infinite."

"This is an attempt to bring the dead back to life," Hyde said. "This is Easter. SALT was murdered by the Soviets when they developed the SS25," an intercontinental missile the administration claims breaks the treaty's limits on new missiles.

Proponents of the resolution argued that Soviet violations of some provisions of SALT II did not warrant a scrapping of the entire treaty.

"The choice is between SALT II or no treaty, between some limits and no limits," said Rep. Lee H. Hamilton (D-Ind.). "We must not throw away an arms agreement we have in hand for a hope of an arms agreement that is beyond our grasp."

Before adopting the resolution, the House unanimously agreed to a Republican amendment supporting U.S. efforts to negotiate a new treaty requiring arms reductions. Earlier, the House rejected, 222 to 187, a GOP substitute measure that critics said would be meaningless because of loopholes.

There is less support in the Senate than in the House for enactment of binding legislation to force continued compliance with the SALT limits. But in addition to the Hart amendment approved by the Senate Armed Services Committee yesterday, a bipartisan group composed of Sens. John H. Chafee (R-R.I.), Dale Bumpers (D-Ark.), John Heinz (R-Pa.) and Patrick J. Leahy (D-Vt.) has introduced a nonbinding resolution similar to the House-passed measure.

The Armed Services Committee yesterday also approved a second Hart measure reaffirming a ceiling of 50 MX missiles that was set in legislation last year.

In a report and letter to Congress yesterday, Reagan said the administration will take into account the "nature and magnitude of the threat posed by Soviet strategic forces" in determining whether to retire older missiles as new missiles are added to the U.S. arsenal.