The Senate finally crushed a stubborn week-long filibuster by a few conservative senators against a nickel-a-gallon increase in the federal gasoline tax last night, but not before the once-mighty, well-oiled Republican leadership machine almost ran out of gas.

After voting 89 to 5 early in the evening to choke off further debate, in what one senator called "trial by ordeal," the Senate began disposing of up to 400 amendments proposed by the legislation's conservative opponents.

Before recessing for the night, the Republican leadership scheduled a series of votes that it hopes will pass the bill by late today.

But the fight over increasing the gas tax to pay for renovating the nation's rundown highways and bridges would not end with Senate passage of the measure, which is backed by the Reagan administration and Republican and Democratic congressional leaders.

There is still doubt about whether the measure can clear other hurdles, including a conference over differences with the House and final approval by both houses, before the 97th Congress finally adjourns, presumably before Christmas.

The extraordinary episode, epitomized by Sen. John P. East (R-N.C.) single-handedly delaying the Senate in the early hours of yesterday morning from finishing work on legislation to keep the government going, raised much larger questions that will haunt the 98th Congress, which convenes next month.

The real issue in the tumultuous fight was not whether to approve the gasoline tax and accompanying legislation for highway construction and changes in trucking regulation, suggested a high-level Republican aide, as the conservative Republican mavericks exploited complex Senate rules to foil their own leadership. "The issue," he said, "is control of the United States Senate."

Some Democrats such as Sen. Paul E. Tsongas (Mass.) said the question was whether the political center is being so "crowded out" by extremes that the legislative process in the Senate has turned into a "trial by ordeal."

In contrast with the impressive discipline which the Senate's Republican leadership used to help hand President Reagan victory after victory at the start of his term, GOP leaders had to use every trick at their command to make possible a vote to end the gas tax filibuster. And they only got it largely because of a tactical error by East.

He had caught the leadership napping at a critical moment late Saturday night and, in effect, grabbed control of the Senate floor at the end of a quorum call. He then held the floor for about four hours as Saturday became Sunday and his fellow Republican senators grew frustrated and angry.

While East talked at length about the regressiveness of the gasoline tax and its impact on North Carolina, other Republicans huddled on and off the floor and milled aimlessly about as scores of spectators in the gallery, including nervous administration officials, watched with rapt attention. Most Democrats simply left the Senate floor, content to let the Republicans squirm.

East, who sat most of the time, alone, in the wheelchair to which he is confined because of polio, was abandoned by other conservatives who had been filibustering the gasoline tax increase.

Finally, by inadvertence or necessity, East appeared to utter the magic words that, under Senate rules, trigger a quorum call, after which the leadership could reclaim the floor. East insisted repeatedly that he had not said the words, which suggested a quorum might not be present, but the presiding officer indicated otherwise.

When East slipped, the leadership, now fully awake, pounced, took back the floor, and mapped strategy for last night's vote to stop the filibuster.

In weary post-mortems, some suggested the embarrassing and humbling episode may turn out to have been a catharsis that will bring Republicans back together in renewed strength and discipline. But others were not as certain, suggesting the glory days for the Senate's two-year-old Republican majority probably have ended as President Reagan's once-tight grip on Congress has loosened.

Attention has focused on the House, only nominally controlled by Democrats during the first two years of the Reagan administration, because the Democrats regained real control by picking up 26 seats in the November elections. But, ironically, the election-induced tremors have so far had a more unsettling effect on the Republican-controlled Senate, where the GOP came out of the election with as many seats as it had before.

Not only have arch-conservatives like East felt free to leave the reservation, bristling because they believe Reagan and Senate Republican leaders have become soft about conservative social issues like abortion, busing and school prayer. But more liberal GOP senators have read the election returns as requiring a more progressive posture for the party.

Almost lost in the spectacle of the conservative rebellion was a more discrete effort by Senate Appropriations Committee Chairman Mark O. Hatfield (R-Ore.) and other progressives in the party to insist that the Senate approve at least some money for jobs creation in its version of the stopgap money bill to keep the government operating.

Over objections from other Republican leaders that even the $1.2 billion proposed by Hatfield's committee, modest by comparison with the $5.4 billion approved by the Democratic House, would be vetoed by Reagan, the Senate approved the jobs money.

A little rebellion is not new to the Senate, which has been straying from the Reagan script with increasing frequency the last year or so. Moreover, the rules have always been there for exploitation.

What is new, especially since last month's elections, is an erosion of the internal self-discipline of the GOP that enabled Majority Leader Howard H. Baker Jr. (R-Tenn.) to run a kind of collegial leadership based on consensus and the common political good of all Republicans.

Baker's style of leading by reason, patience and good will, rather than the force and arm-twisting tactics of a Lyndon B. Johnson, is ideally adapted to good times when everyone stands to profit politically by going along with the pack. It appears bound to be more severely tried in bad times.

It is not by happenstance that many of the newly wayward Republicans are those who are up for reelection in 1984, when the GOP will be defending more Senate seats than the Democrats will for the first time in many elections. This not only poses problems for individual senators but threatens Republican control of the Senate, which compounds Baker's problems as he seeks to restore discipline.

Yet, for all his laid-back style of leadership, Baker demonstrated over the last couple of days that he is not without resources to continue winning. His polite treatment of the Democrats over the last two years paid dividends when they rallied, with some reluctance, behind his effort to break the filibuster against the gas tax bill. Moreover, he demonstrated he can play hard-ball by embracing rarely used, unusually tough tactics to break the gas tax filibuster.

As a result, the only senators who voted against ending the filibuster last night were Republicans East, Jesse Helms of North Carolina, Gordon J. Humphrey of New Hampshire and Don Nickles of Oklahoma, and Democrat William Proxmire of Wisconsin.