When it was coming to an end early Friday morning and everyone knew that the House of Representatives was going to say no to the last remaining option on a federal budget for next year, the members grew strangely quiet.

As a rule, House sessions that run past midnight--as this one had done an hour earlier--are what reporters euphemistically call "boisterous." But Minority Leader Robert H. Michel (R-Ill.) was not embroidering when he said, in closing debate, that the House could be proud that after being in session on the budget for 30 of the previous 39 hours, the mood was "as orderly as it is."

The situation was sober--even if the hour was not. In rapid-fire fashion, the House had chopped down Michel's budget, backed by President Reagan, a counter-budget offered by a bipartisan group of younger members, and the Democratic budget crafted by Budget Committee Chairman James R. Jones (D-Okla.).

Now, all that was left was Jones' unamended version, and that, too, was foredoomed to defeat as soon as Michel sat down. The one thing almost everyone in leadership, Republican or Democratic, had wanted to avoid--the thing the amazingly open rules of debate were designed to prevent--now loomed. The House was going to tell an anxious financial community and a recession-wracked country that there was no consensus for any blueprint of national fiscal policy for the next fiscal year.

There was almost a note of despair in the voice of House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.). "I say to my liberal and conservative friends," he said, sweeping the House with his gesture, "I don't know how you get together."

Budget-making has been tough work in the House since the new congressional budget process started in 1974. "It's always the parts against the whole," said a Democratic leadership aide. House members thrive by championing particular projects, whether they be weapons systems, agricultural laboratories or education grants. Weighing the competing claims of those programs against each other, as the budget process forces them to do, is politically and personally painful.

The House budget process has always been more partisan and less accommodating than the Senate's. Twice before--in 1977 and 1980--the House was unable to assemble a majority for any budget the first time around. But never had there been the emphasis of this year on the psychological and practical importance of writing a fiscal blueprint that realistically promised smaller deficits. Otherwise, almost everyone said, there was little hope of reducing the high interest rates that are choking the economy.

Recognizing that imperative, Rules Committee Chairman Richard Bolling (D-Mo.) devised a procedure for debating and voting that gave every group in the House a chance to test its pet proposals before the final, major alternatives came to a vote.

The Bolling rule was accepted by both parties and all factions, encouraging hopes that it would succeed in its goal: getting a budget passed.

Said one man instrumental in devising the plan: "Dick deliberately borrowed the Senate technique. He gave every member a chance to vote this way and that way. He let them go up the hill and down the hill, all in hopes that eventually they'd vote for a bill."

But it didn't work out that way. There were 30 roll calls on amendments before the final voting started; seven amendments passed and more prevailed by voice vote. But in the end, the majority still said no to any bill.

Why did it fail? As O'Neill said before the final vote, "There is blame enough to go around."

Start with the economy itself. The combination of high unemployment, high deficits and high interest rates is an intimidating one. Few in the House had any real confidence they knew the right answer for the triple puzzle. Rep. Barber B. Conable Jr. (R-N.Y.), a supporter of Michel's budget and a widely respected voice on economic policy, said early in the week, "I'm not sure any one of these budgets is an automatic corrective for our economic ills; morale is so low in the financial community, none of these may help."

More pungently, a lobbyist off the House floor remarked, "Winning this thing is like winning the Falklands. What the hell have you won when you've won?"

Last year, the budget fight in the House was bitter and partisan, but the economic mood was much brighter. President Reagan asked Congress to do things that, in retrospect, look easy: vote for big tax cuts, a stronger defense and only a few snips in the safety net.

This year, the fiscal 1983 budget Reagan laid on Congress' doorstep was referred to even by Republicans as conservative as Rep. Mickey Edwards of Oklahoma as "a turkey." Rejected in its original form by the Senate Budget Committee, it had few enthusiastic boosters in the House. Conservatives abhorred its $100 billion deficit, and liberals objected to its heavy defense spending and domestic service cuts. Michel complained that some of the domestic cuts were "politically stupid," and Reagan himself seemed notably muted in his support.

"Yeah, he called me," said one Republican who was the recipient of a presidential phone call, "but it sounded like he was just reading his lines."

The GOP never developed the momentum and drive for the budget that it displayed in the springtime of Reagan's presidency last year.

On the Democratic side, as usual, the leadership was beset by divided counsel. Some--like Jones--wanted most of all to avenge last year's defeat and show that the House would pass a Democratic budget. They urged a strategy of concessions, aimed at beating the plan sponsored by Michel and Rep. Delbert L. Latta (R-Ohio).

Others preferred to have the budget--and the economy--as a 1982 election issue. They were hoping, not very privately, that Michel would prevail. Why put the Democrats' fingerprints on the budget at this point, they asked. Let the Republicans have the responsibility, and the blame.

In the end, their own division kept the Democrats from gaining either goal. They neither won with a budget of their own nor pinned the blame on the Republicans. Instead, they handed the GOP a new issue: the disarray in the Democratic House.

In retrospect, most observers agreed with Michel's assessment that his modified version of the Reagan budget had the best chance to pass. At the last moment, Democrats decided not to offer a Medicare "sweetener" amendment to that budget, because, as O'Neill said, "there was no question that if it passed, the Republican Michel-Latta budget would pass."

Instead, they offered a stiffer amendment, boosting Medicare even more, but taking a commensurate amount out of defense. With that unwanted rider aboard, Michel was unable to attract enough pro-defense conservative Democrats to offset the defections of liberal Republicans who objected to the remaining domestic cuts.

Still, Michel, with 192 votes, came closer than anybody to winning and he has the best chance of assembling a majority when the House tries again to construct a budget, beginning next week.

Meantime, the complex, fascinating legislative whirl revealed new strains and coalitions in both parties.

On the Republican side, some 60 conservatives signaled their unhappiness with both the Reagan administration and with Michel, by voting "present" until the final seconds of the key Medicare roll call. That group, which wants even deeper domestic cuts than Reagan, to speed a balanced budget, strengthened its bargaining hand for the second round of the battle, as Michel conceded when he said Friday he would try to shift "a few degrees" to the right.

But concessions to them will further alienate the two dozen moderate Republican "Gypsy Moths," already feeling politically exposed this election year because of their past support of Reaganomics. Many are off the Reagan train, at least until November.

On the Democratic side, the budget debate exposed still new fissures. In addition to the "Boll Weevil" conservative southern defection, which once again cost the party some 30 votes, there was an angry rebellion by most of the 18-member Congressional Black Caucus. Angered because they felt their own budget proposals were brushed off by the party leadership, 13 of the 18 voted against the Jones budget, a stand that was echoed by about a dozen white liberals.

That double defection, at both wings of the party, left Jones still frustrated in his effort to pass a Democratic budget. Since winning the Budget Committee chairmanship by a single vote in 1980, Jones has yet to prevail on the House floor, and his political standing as a future aspirant for higher leadership positions has suffered accordingly.

But he was not the only "future star" to have his light dimmed last week. Five other young Budget Committee Democrats, who think of themselves as post-Great Society liberals, hooked up with a handful of liberal Ripon Society Republican representatives to concoct a "future-oriented" budget, emphasizing capital investment, education and training, technology and defense modernization--not supply-side tax cuts or the old welfare programs.

They got only 137 votes for their plan, indicating the House is far from ready to endorse their thinking.

In fact, about the only people left relatively unfazed by last week's failure were some very old House members, who think nostaligically of the power they enjoyed before the 1974 budget process came along. Appropriations Committee Chairman Jamie L. Whitten (D-Miss.), who came to the House in 1941, put through an amendment late Thursday that would have ended one of the key enforcement tools of the budget process.

That provision blocks any individual appropriation bill until Congress has agreed on a final budget for the year. Although the Whitten amendment died with the rest of the measures voted down early Friday morning, it was taken as a serious signal of growing House impatience with the whole budget process.