President Reagan suffered another embarrassing setback on defense yesterday as the Republican-controlled Senate Appropriations Committee voted, 16 to 12, to bar spending of MX production funds until both houses of Congress approve a plan for basing the controversial missile.

The vote came on an amendment by Sen. Ernest F. Hollings (D-S. C.) to a stopgap funding bill that must be passed to keep the government from shutting down at midnight Friday.

The Hollings amendment would postpone the spending of $988 million in MX production funds in fiscal 1983, but leave intact $2.5 billion for MX research and development.

It would force the administration to reevaluate its Dense Pack plan for bunching 100 MXs in superhardened silos in Wyoming. Reagan made much in the 1980 campaign of the failure of his predecessor to come up with a deployment plan under which MX would be relatively invulnerable to attack. His answer to this problem--Dense Pack--is based on the theory that incoming Soviet missiles would also have to be bunched and would be destroyed by their own blast. Critics have laughed it away as unworkable.

Four Republicans voted with Hollings and against the president yesterday: committee Chairman Mark O. Hatfield (Ore.); Lowell P. Weicker Jr. (Conn.), Mark Andrews (N. D.) and Arlen Specter (Pa.). The only Democrat to vote with the president was John C. Stennis (Miss.).

"Dense Pack is dead," a jubilant Hollings said after the vote. "This is a plain and simple victory for common sense," he said, adding that if Congress ever approves a new land-based missile, it will probably have to be smaller and more mobile than MX.

But the issue now goes to the Senate floor where MX supporters will propose a face-saving substitute endorsed by President Reagan, to block the spending, but require Congress to vote up or down on a basing plan by April 15.

At the other end of the issue, Sens. Alan Cranston (D-Calif.), Charles McC. Mathias Jr. (R-Md.) and 13 co-sponsors said they will try to cut the 1983 funds unconditionally, as was done in the House.

Reacting to the appropriations vote, White House spokesman Larry Speakes warned that if the Hollings amendment became law, "Congress could postpone indefinitely a vote on a new missile and leave uncertain the question of America's resolve to rebuild its national defense."

"Our negotiations at START Strategic Arms Reduction Talks would be substantially weakened in their pursuit of an arms control agreement," he said.

However, Cranston called the administration's alternative "merely a delaying tactic. We've already spent $4 billion in 10 years trying to come up with an invulnerable basing mode. It has become unmistakably clear that none will work."

White House officials said last night that Reagan, Vice President Bush, arms control negotiator Edward Rowney and other administration officials had called committee members yesterday to push the so-called fast track alternative requiring a vote by April 15.

However, that alternative never surfaced in committee. Sen. Ted Stevens (R-Alaska) told committee members he was unprepared to deal with the issue in committee and preferred to put it off until floor debate. When Hollings refused, Stevens offered a motion to table Hollings' amendment. It was on that tabling motion that the 16-to-12 vote came.

Talking with reporters after the vote, a somewhat flustered Stevens said he had told Reagan that morning that the issue would not come up in committee. "I was not prepared," he said. "It was my error, that's all . . . . I never had any notice it was going to come up."

Hollings scoffed at that. "Sure he was prepared," he said. "He was being disingenuous. He did not have the votes."

Stevens said he was confident the administration's language forcing an early vote would be adopted on the floor.

And Hollings said that while he has opposed it so far, in order to keep Hatfield and other MX opponents on board, he would support an accelerated timetable on the floor. This means Reagan may be able to claim a partial victory there, though the whole issue then will be subject to further discussion in conference with the House, which killed the production money outright.

Another controversial defense issue was defused yesterday when the committee, which had originally voted to reduce European troops to 1981 levels, in a move to force allies to take more responsibility for their defense, decided to cap them at 1982 levels instead. That cap would not require immediate reductions.

A House-Senate conference will have to resolve the differences over the MX, over the troop levels -- there are no House restrictions -- and over the total defense spending figure. In addition, the House has cut production funds for the Pershing II missile, scheduled to be deployed in Europe next year, but the Senate has retained the funds.