The Democratic-controlled House yesterday broke its habit of rubber-stamping President Reagan's budget plans and, defying veto threats, refused to approve big new cuts in domestic appropriations for the rest of the fiscal year.

But Reagan won in the Republican Senate as its Budget Committee rejected plans to move ahead immediately with tax increases and entitlement program cuts that Reagan doesn't want at this time.

Voting 201 to 189, the House spurned a proposal by Minority Leader Robert H. Michel (R-Ill.) for a 5 percent domestic spending cut in a catchall bill to fund the government after its current stopgap funding authority runs out on Friday.

Michel said he was pushing for the cut "to avoid a presidential veto," and Senate Republicans are preparing a similar effort for later in the week in hopes of avoiding a veto impasse, which could leave the government without funds to operate next week.

The so-called "continuing resolution," which was required because Congress has yet to adopt any of its 13 appropriations bills, was given final House approval, 195 to 187, in an atmosphere of confusion and conflicting claims about whether Reagan and the House were actually all that far apart on the spending question.

Last September, alarmed at mounting deficit projections, Reagan proposed a 12 percent cut from the domestic appropriations levels that he had originally proposed in March. Reagan's Office of Management and Budget claimed yesterday that the continuing resolution reflected only a smidgen of those cuts, falling $10.4 billion short of the savings he wanted.

But, using figures from the Congressional Budget Office, House Democrats insisted that the resolution fell only $2 billion short of Reagan's goal for domestic spending cuts and, when defense cuts proposed by a House Appropriations subcommittee were included, actually exceeded Reagan's proposed savings by more than $4 billion.

The Democrats claimed this was accomplished because spending anticipated by the resolution would be set at the lowest of several possible levels, including those now in effect as well as new levels approved in 1982 appropriations bills that are now moving through Congress. But OMB contended that some savings claimed by the House Democrats couldn't be counted toward Reagan's goal, and there were other disputes as well.

The situation left the Democrats in a somewhat embarrassing position, even though they could claim their first major budget victory over Reagan, who forced $35 billion in spending cuts through both houses of Congress.

"Most of us had been arguing that the 12 percent cuts were egregious," conceded House Budget Committee member Leon E. Panetta (D-Calif.), who said the Democrats themselves were shocked to discover from CBO over the weekend that spending levels contained in the resolution were so low.

"I'd be happier if I knew where these cuts were coming from," complained Rep. Les Aspin (D-Wis.), another Budget Committee member, after spending a couple of largely futile hours with OMB officials just off the House floor trying to sort out what amounted to an $8 billion discrepancy in OMB and CBO figures.

The 201-to-189 vote against further cuts in the continuing resolution reflected erosion of the Republican and conservative Democratic "boll weevil" coalition that gave Reagan his impressive victories in previous House votes.

Only 29 Democrats supported Michel's proposal, and 18 Republicans voted against it. The vote among Washington-area members was along party lines, with Republicans supporting Michel and Democrats opposing him.

As approved, the continuing resolution confronts Reagan with nearly twice the $2 billion in defense cutbacks he wanted from the big military buildup he has recommended for 1982 and beyond. But the figure is subject to further action this week in both the House and Senate.

While Michel would have exempted military and some other programs from his proposed cut, the Senate Republicans are considering a plan that would cut 2 percent from all programs, including defense. Still at issue late yesterday was whether it would be an across-the-board cut or whether Reagan would be given limited impoundment authority to make the cuts as he chooses.

The resolution carries a price tag of $440 billion to fund the government through the end of the fiscal year on Sept. 30, 1982, although the size will shrink as individual appropriations bills are enacted.

In the Senate, the Budget Committee failed on three successive votes to find a majority willing to support an assortment of tax increases and spending cuts that could be combined into a second budget resolution for the current year. That was a victory for the Reagan administration, which had urged putting off any such package until Reagan comes up with new budget proposals in January.

Budget Committee Chairman Pete V. Domenici (R-N.M.), who was bucking Reagan and Senate Majority Leader Howard H. Baker Jr. (R-Tenn.) in pushing for action now, saw his own plan fail on a vote of 12 to 10. Its main features were $48 billion in tax increases and $40 billion in cuts from federal entitlement programs by 1984 in order to balance the budget in that year--a goal that Reagan has now abandoned.

Democrats punched holes in the Republican chairman's plan, contending he exaggerated the amount of savings to be squeezed out of entitlement programs. "I don't think your package is ready to fly," said Sen. Lawton Chiles (D-Fla.).

After the voting, Domenici told reporters he is not sure what his committee's next move will be. Passage of some kind of second budget resolution is required before Congress can adjourn for the year next month.

Domenici said one alternative for the committee would be to do what the House Budget Committee has already done, which would be to merely extend the year's first budget resolution and wait for Reagan's January budget message before making real changes.

Even before yesterday's vote, despite strong Republican support on the committee for Domenici's plan, few in the Senate or House thought Domenici would be able to get a plan out of his committee and get it approved on the Senate and House floors.

An alternative to Domenici's plan submitted by Sen. Ernest F. Hollings (D-S.C.) was beaten by a vote of 16 to 5. It would have stressed large tax increases to wipe out deficits by 1984. Another plan, proposed by Sen. Howard M. Metzenbaum (D-Ohio), lost by a large margin.