In a dramatic late night session, the House yesterday endorsed President Reagan's nuclear weapons control policy by a 204-to-202 vote.

The approval of an administration-sponsored resolution calling for reductions in nuclear weapons--but not for an immediate arms freeze--was a setback for the nuclear freeze movement, which has sought to repudiate Reagan's plans for a massive weapons buildup to match the Soviets.

Reagan called wavering Republican House members on the telephone during the eight-hour House debate, arguing that the nuclear freeze resolution sponsored by House Foreign Affairs Committee Chairman Clement J. Zablocki (D-Wis.) would hamper arms negotiations underway in Geneva.

Rep. Edward J. Markey (D-Mass.), a leader of the congressional freeze movement, predicted that Reagan's "arm-twisting will result in political peril for a large number of congressmen. When we come back here next year, we will have the votes not just to pass a nuclear freeze resolution, but to defeat first-strike destabilizing weapons" such as the MX and Trident II missiles.

While the close vote last night reflected the growing strength of the freeze movement, freeze advocates acknowledged that the vote on the non-binding resolutions was to some extent symbolic compared with House approval last week of a $175 billion defense authorization, the largest peacetime increase in history.

Approval of the administration-endorsed resolution, sponsored by Rep. William S. Broomfield (R-Mich.), came after a dramatic speech by House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.), who rarely takes the floor.

The chamber, which had been filled with members chatting in the aisles, fell to a hush as O'Neill recounted how he personally had witnessed explosions of two atom bombs in Yucca Flat, Nev., 30 years ago.

"The world is racing toward catastrophe," O'Neill said. "Something must be done . . . . The brakes have not been put on. Instead, the foot is on the accelerator, and now we're about to put the accelerator to the floor with a massive buildup in strategic weapons."

As the electric clock overlooking the chamber ran out of time, the vote seesawed between victory and defeat for each side. When it was 198 to 200 the Democrats on the right side of the chamber rose to cheer. Seconds later, it flipflopped to 200 to 199, and Republicans on the left side jumped to their feet, whistling and clapping.

Then, as the scoreboard registered 202 to 202, Lawrence Coughlin (R-Pa.), who had been called by the president a few hours before, switched his vote to "aye". O'Neill voted "nay." Another Republican voted "aye" and the chamber erupted in shouts and wails.

Zablocki's freeze resolution called on the administration to pursue "a mutual and verifiable freeze" on nuclear weapons by the United States and the Soviet Union, followed by reductions in nuclear arsenals.

The Broomfield alternative called for "an equitable and verifiable agreement which freezes strategic nuclear forces at equal and substantially reduced levels."

The Senate Foreign Affairs Committee has reported out a resolution similar to the Broomfield version, although it also calls on the president to adhere to the Strategic Arms Limitation Treaty, SALT II, negotiated by President Carter but unratified by the Senate.

Arguing for his "freeze now--reduce later" approach, Zablocki said, "The grass roots have spoken. People want to see this nuclear buildup stopped. The genie must be put back in the bottle."

Broomfield said the Zablocki resolution would "cripple" arms control talks now under way in Geneva.

"Is the Soviet Union likely to negotiate seriously with an administration whose position has been repudiated by this Congress?" he asked, echoing the feeling of many Republicans that the administration's policy and international prestige was at stake.

Referring to Democratic threats to use the freeze issue against Reagan and GOP candidates, Broomfield said the Zablocki resolution "may be good politics for some, but it is bad arms control policy."

Rep. Henry J. Hyde (R-Ill.), arguing that a freeze would lock the United States into an inferior arms position vis-a-vis the Soviets, said, "It's like freezing with their hands at our throat. I'd rather put their hands down and then freeze." Zablocki's resolution, he said, "is government by bumper sticker."

Republicans, particularly from the Northeast, where the freeze movement is strong, said they were under "strong pressure" from the administration to vote against the Zablocki resolution.

Reagan called Rep. Christopher H. Smith (R-N.J.), urging him to vote with Broomfield. He did not. Rep. James K. Coyne (R-Pa.) had said he would vote with Zablocki in spite of a call from Bush, but ended up voting for Broomfield.

Rep. Judd Gregg (R-N.H.) said 40 or 50 calls urging him to support Zablocki poured into his office from his district yesterday. He voted against Broomfield.

The vote was largely along party lines, with 151 Republicans and 53 Democrats voting in favor of the Broomfield resolution, and 175 Democrats and 27 Republicans voting against it. The entire Virginia delegation voted for it, while Reps. Michael D. Barnes and Steny Hoyer (D-Md.) voted against it. Rep. Marjorie S. Holt (R-Md.) voted in favor.

Reagan, accompanied by Bush, Secretary of Defense Caspar W. Weinberger, Secretary of State George P. Shultz and top White House aides, reportedly turned several votes around yesterday in a meeting with 10 House members in the Cabinet room.

To win last-minute votes, freeze advocates had eliminated Foreign Affairs Committee language calling for prompt approval of SALT II. Instead, the final Zablocki resolution said the United States "shall continue to adhere to the Salt II agreement" so long as the Soviet Union does and it is in the national security interests of the United States to do so.

Freeze proponents argued that the Reagan administration's move to build more first-strike weapons such as the MX and Trident II missiles--programs which the House approved last week in the 1983 defense authorization bill--would cause the Soviets to respond in kind, thus accelerating the arms race.

"Both sides can now reduce the other to rubble," Markey said. A freeze, he added, would "bring the arms race to a grinding halt" while the Broomfield approach would "continue the arms race to achieve some mythical level of equality."

Rep. Jonathan B. Bingham (D-N.Y.), contending that the United States' new weapons systems will increase the likelihood of an accidental nuclear war, said, "The beauty of the freeze idea is its common sense, its simplicity. The way to stop the arms race is to stop it."