An uneasy House yesterday tied some strings to the MX victory it handed President Reagan Wednesday night, conditioning deployment of the missile on further steps toward arms control.

In the Senate, meanwhile, Republican leaders failed by five votes to choke off debate on the MX and force it to a vote, although efforts were under way last night to negotiate a vote Tuesday.

In separate voice votes, the House adopted provisions to reduce the number of MX missiles the administration can buy next fiscal year and to make their deployment contingent on progress toward a smaller, less-threatening alternative missile in the future.

At the same time Senate leaders failed in a first effort to curtail the anti-MX campaign by Democratic presidential hopeful Sen. Gary Hart (D-Colo.), as only 55 senators, five fewer than needed, voted for cloture.

But Senate Majority Leader Howard H. Baker Jr. (R-Tenn.) scheduled a second cloture vote today unless negotiations between leaders of both parties, including Hart, produce an agreement on scheduling debate and votes through next Tuesday.

The House disposed of all the MX issues on its plate yesterday afternoon in an unsuccessful attempt to complete action on the $187.8 billion defense authorization bill that it took up in mid-May, and which it may not finish until September.

Rep. Les Aspin (D-Wis.), a key broker of the plan that led to House approval of $2.5 billion in MX production funds Wednesday night, followed up yesterday with an amendment aimed also at guaranteeing development of a small, mobile intercontinental ballistic missile system that might one day replace the MX.

President Reagan's bipartisan Commission on Strategic Forces recommended this spring that Congress let the MX go forward but that the administration also begin work on a smaller missile for the future, called Midgetman, and take other steps toward arms control.

The House, which earlier had blocked the MX, bought that compromise in May, but some members remained fearful that the Midgetman would fall by the Pentagon wayside once funds for the 10-warhead MX were nailed down.

Aspin's amendment, accepted reluctantly by House Republicans, would limit MX deployment to 10 missiles until a Midgetman prototype has been tested and then to 40 missiles until the Midgetman has been flight-tested and contracts signed for full-scale engineering development.

The Pentagon also must attempt to keep Midgetman a small missile weighing no more than 33,000 pounds.

The House yesterday also approved an amendment by Reps. Albert Gore Jr. (D-Tenn.) and Norman D. Dicks (D-Wash.) to cut the MX authorization voted Wednesday night by more than $350 million and reduce the number of MX missiles procured in the coming year from 27 to 21.

Gore said it was also part of a plan to make sure that the number of MXs to be deployed in existing Minuteman silos will not, in combination with other weapons systems such as the Trident II, give the United States "a first-strike capability."

The administration has proposed deployment of 100 MXs in the next several years; Gore would like to limit this to 50.

Gore said the MX production lid should save $2.3 billion over the next six years. He described it as part of the bargain that he and other House Democrats struck with the administration in return for their support of the program.

He said that by his calculations, deployment of 100 MXs, in conjunction with other weapons, would give the United States a menacing "first-strike capability" to knock out Soviet ICBMs in their silos.

Reagan has urged approval of the MX "not only for force modernization, but to keep the Soviets at the negotiation table."

Critics of the big weapon, however, have vowed to keep trying to kill it despite the 220-to-207 House vote Wednesday night to authorize production. Another crucial test will come this fall when Congress must decide whether to appropriate the procurement money.

Gore agreed in an interview that the slender 13-vote margin for the MX was a clear sign of the fragility of the pro-MX coalition in Congress. "Measurable movement on arms control is the only thing that will stave off defeat in the fall," he said.

The Senate vote to shut down a rambling two-week debate over the MX and other elements of a $200 billion defense authorization bill for next year was 55 to 41.

If cloture is voted today and anti-MX senators are given ample time to air their views, Hart said there is a "strong possibility" that the MX foes would allow the Senate to begin voting at a "reasonable time" on MX-related amendments.

But, as a hedge, Hart has filed more than 500 amendments that could be used in a post-cloture fight to stall action on the bill.

Facing continued deadlock, the two sides began trying to wind down the quasi-filibuster, and Baker, who had earlier indicated optimism about cloture today, said he was fairly confident a good agreement could be worked out overnight.

Hart and his allies have been holding up votes since late last week on grounds that the GOP leadership has thwarted a true debate, not only by refusing to respond to criticism of the MX, according to Hart, but also by delaying MX opponents when they want to give speeches.

In yesterday's cloture vote, two Republicans, Bob Packwood (Ore.) and Arlen Specter (Pa.), joined all but six Democrats in voting to limit debate. The six Democrats were Dennis DeConcini (Ariz.), J. James Exon (Neb.), Howell Heflin (Ala.), Henry M. Jackson (Wash.), J. Bennett Johnston (La.) and Edward Zorinsky (Neb.).

But seven of the senators who joined Hart in opposition to cloture supported the MX in an earlier vote in May, raising doubts about how long Hart can count on their support.

In action on the defense bill yesterday, the Senate voted 56 to 40 for a proposal by Sen. Daniel Patrick Moynihan (D-N.Y.) to give students and colleges another month, until Sept. 30, to comply with a law requiring recipients of federal student aid to register for the draft.