The House Budget Committee called President Reagan's bluff yesterday and voted to send his original budget to the House floor, even as members of both parties conceded that it would not pass and scrambled to put together rival alternatives.

The committee made clear that it was reporting out Reagan's budget only as a vehicle for consideraation of two alternatives, one each from Republicans annd Democrats. Only if both alternatives fail would Reagan's budget come up for a vote.

"We are reporting the president's budget to the floor for the single reason that the president continues to state that the House has denied his budget this consideration," House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) said. "By making the president's budget the formal vehicle next week, we hope to expedite a final agreement."

But, with the Reagan budget's deficit projected at $122 billion for next year, considerably more than that in any of the three major budget proposals rejected by the House last week, at least some Republicans were nervous about what might happen if the other proposals failed and the president's proposal came up for a vote.

"It could be quite embarrassing not only to the president but to the country," especially if Reagan's budget were repudiated while he was attending the economic summit at Versailles, Rep. Leon E. Panetta (D-Calif.) also questioned the wisdom of using Reagan's budget as a vehicle. "It was dead six months ago, it is dead today and it will be dead when it is brought up on the House floor," he protested. "There is a very real possibility we will end up with nothing."

Republicans, however, appeared determined to produce a winning budget this time, even if it means some ideological impurity, and many Democrats conceded that the GOP has the best shot at passing a budget.

"We're playing the margin game out there to get 218 votes," a majority of the House, Minority Leader Roberty H. Michel (R-Ill.) said after a party caucus in which sentiment appeared to run against a conservatives' hard-line proposal to freeze most domestic spending and run for restoration of some domestic spending.

One plan under cnsideration would restore $2.5 billion for federal pay and pensions, Medicare, Unemployment benefit and other programs, while cutting foreign aid by $500 million and anticipating $4 billion more in revenues, leaving a deficit of just over $100 billion.

Many of these proposals reflected amendments approved by the House last week before it rejected all the budget proposals.

But the new GOP draft is subject to change after head counts indicate what specific modifications may be necessary to gain enough votes to win, GOP sources said.

The tenor of yesterday's caucus pleased some moderates, who had been worried about earlier talk of moving even farther to the right than the GOP proposal defeated last week. But it left some conservatives unhappy.

"For 90 percent of those people, this a seat in the House is the best job they ever had and they want to hold onto it," conservative Rep. Eugene Johnston (R-N.C.) complained as he left the caucus. "They're writing a political document, not an economic document."

"I think there is reason to believe they're fine-tuning it their budget proposal of last week in a direction that would be helpful to us," said Rep. James M. Jeffords (R-Vt.), a moderate.

As before, Democrats are divided between those who see political profit from going down to honorable defeat--"death with dignity," one aide calls it--and those who want to win. But, as several Democrats explained yesterday, the current strategy involves a fight to the finish but in a way that leaves the Republicans no grounds for complaint if they lose, paving the way for a compromise.

Budget Committee Chairman James R. Jones (D-Okla.), shuttling between O"Neill and committee Democrats with lists of proposals, was reportedly considering many of the same "sweeteners" that the Republicans are exploring and others. Still reportedly unresolved last night was how to offset the cost.

Earlier in the day, O"Neill said he is determined that the new Democratic proposal include funds for unemployment compensation, emergency jobs and housing, as provided in the earlier Democratic proposal, along with restoration of Medicare and education funding approved by the House last week.