Despite veto threats, the Senate by a decisive 70-to-23 vote refused yesterday to kill a $5.1 billion housing stimulus plan opposed by President Reagan. The plan could subsidize 4 percentage points on a homeowner's mortgage.

The president in recent days has repeatedly indicated he will veto the urgent supplemental money bill, to which the housing plan would be attached, because he views it as a special-interest "bailout" for a single industry.

But the Senate rejected the move of Majority Leader Howard H. Baker Jr. (R-Tenn.) to strip from the bill a revised version of the housing package sponsored by Sen. Richard G. Lugar (R-Ind.).

Although yesterday's vote is not final -- the bill faces a stalling action from Sen. William L. Armstrong (R-Colo.) and the Lugar plan has not yet been nailed into the language -- it was the first clear test of sentiment on the proposal.

Under the new Lugar version, the government would make commitments by the start of November to subsidize interest rates on 400,000 new low- and moderate-income homes. The total cost would be $5.1 billion in government payments over the next five years. The Lugar amendment would appropriate $1 billion in fiscal '82 as the first installment.

The earlier Lugar version, approved by the Senate Appropriations Committee when it supported the supplemental bill, would have permitted commitments on only 80,000 units this year and the rest in later years. The House has already passed a bill calling for about 74,000 commitments this year.

Lugar, in an impassioned plea for his program, said it would stimulate the start of construction on new homes, thereby putting 700,000 men to work and helping lead the country out of the recession. He said the full 400,000-unit package must be committed this year in order to get work going again on a major scale in one of the nation's most important industries.

"It is not a bailout, it is not a budget buster," he said, referrng to the just-passed Senate budget resolution that allows for the $1 billion in appropriations required for the bill. And, he added, "it is not a confrontation with the president of the United States."

Sen. William Proxmire (D-Wis.), agreeing, said the Senate Banking Committee had voted 14 to 1 to allow the full 400,000 commitments this year. "We just can't hunker down" and permit "absolute disaster" in the housing industry, said Proxmire.

But Armstrong, who faces a debate-limiting cloture vote today to cut off his stalling tactics, said it would be a "serious mistake" to pass the bill. He said "many homebuilders are skeptical" that the Lugar plan will really stimulate as much construction as Lugar Predicts.

Lugar told the Senate that since the housing plan has provisions for payback of the interest subsidies in later years by the beneficiaries, the ultimate cost to the government would be extremely small. "The bill calls for recapture," he said.

Baker earlier this week had proposed stripping the housing program and several other items out of the supplemental money bill in order to speed action on those money items remaining in the bill that he said are truly urgent, namely funds to allow the treasury to keep writing Social Security, veterans' and some other benefit checks and to permit certain programs to continue and others to avoid having to furlough employes.

Armstrong has said he will not stop talking and stalling until the housing provisions are transferred to another bill for separate consideration.