It was a critical moment for the Senate last month as it struggled with a major civil rights bill, a half-trillion-dollar appropriations bill and a partial government shutdown that was threatened by an impasse over the two measures.

So what was "the world's greatest deliberative body" doing?

A notice from the office of the Senate Democratic whip said it all: "At approximately 2:30 p.m., a vote will occur on the motion to reconsider the vote by which the Senate tabled the appeal of the ruling of the chair that Amendment No. 5728, relating to school busing, was not germane."

Strange as it may seem from this confusing language, the vote was pivotal, paving the way for defeat of the civil rights measure and passage of the appropriations bill, which in turn made it possible for the government to keep spending money and for the 98th Congress to adjourn.

But it was, as one senator observed in the broader context of the Senate's overall behavior this fall, the "twilight zone of the legislative process."

In their own way, the words from that Oct. 1 Democratic whip notice are an apt metaphor for the Senate as its Republican majority prepares to elect new leaders Wednesday and braces for a fight with the Democrats for control of the chamber in 1986.

Often the Senate is inexplicable. Just as frequently it is incapacitated. More and more, senators worry that it is becoming increasingly irrelevant.

"We are witnessing the atrophy of the Senate . . . ," said Sen. Dan Quayle (R-Ind.), chairman of a select committee appointed earlier this year to help rescue the Senate from itself.

"The Senate is facing an identity crisis, and it's at an acute stage," said Norman J. Ornstein, a respected specialist on Congress who is with the American Enterprise Institute.

"It was never supposed to be efficient; it was supposed to be creative," Ornstein said in an interview. "But it isn't creative; it's scattershot. It doesn't just bend over backwards to protect the intense feelings of its minorities, it lets individuals run roughshod over any semblance of institutional process. It has neither a broad perspective nor a narrow capture of detail."

Senators' almost universal sense of dissatisfaction, coupled with the advent of new leaders and an expected outpouring of reform proposals when Congress returns for business in January, could provide a turning point for the troubled institution.

But few expect miracles.

Democrats and Republicans say the old leadership, especially retiring Majority Leader Howard H. Baker Jr. (R-Tenn.), did about as well as possible under the circumstances. And previous efforts at rules reform have been ignored or have spawned new procedural abuses. Some say pressure from voters is the only answer, but they also wonder if voters care whether the Senate functions.

Because it was meant to be more creative than efficient, and to run against the tide of democratic rule by affording one last opportunity to slow or stop a willful majority, streamlining efforts are undertaken with great care by those who most cherish the institution and its traditions.

But even they say the balance has tipped too far toward chaos.

There are no stronger critics than its most devoted members. Respected veterans such as Baker and John C. Stennis (D-Miss.) have spoken with no less passion than younger members such as Quayle about the frustrations of serving in today's Senate. Baker went so far as to quit.

The senators complain of outmoded, loophole-ridden rules and a membership that exploits them to pursue their own political agendas, oblivious of the need for consensus and compromise.

"The balance between institutional and individual rights is out of kilter," Majority Whip Ted Stevens (R-Alaska) said. "Much useful legislation cannot be considered because the rules permit too many barricades. Irrelevant issues are raised through nongermane amendments, forcing a diversion of the Senate's time and energy . . . . The Senate rules have been bent, stretched and interpreted to a point where they now constrain the Senate from doing . . . business."

Members load up appropriations bills with extraneous amendments for the sake of their own "political therapy," complained Appropriations Committee Chairman Mark O. Hatfield (R-Ore.), and "this does violence to the institution."

Not only does the Senate no longer fulfill the founders' vision of a forum for great debate over national policy, the critics say, but it can barely function on a day-to-day basis as it lurches from one procedural quagmire to another.

As for the routine conduct of business, Sen. Bob Packwood (R-Ore.) said, "We have managed from roughly 1970 or 1971 onward to so pervert and torture the processes of this body that we are approaching being inert."

Although it was always meant to be less structured and more hospitable to minority viewpoints than the House, the critics add, the Senate has become almost anarchical in its deference to individual whim and demand, gradually eroding its role as an equal to the House in the congressional system.

"In trying to accommodate everyone, we accommodate no one," said Sen. Mark Andrews (R-N.D.).

Baker has said leading the Senate often can be like trying to "push a wet noodle."

For rank-and-file members who plan to stay around for a while, the frustrations perhaps are even greater. "My personal fear is that we're becoming an anachronism," said Sen. Warren B. Rudman (R-N.H.). "If we're going to become a House of Lords, at least we should get peerages."

Some senators and outsiders worry that the Senate's problems are undercutting its ability to attract good new people.

Especially for those with presidential ambitions, "the Senate is no longer necessarily the place to be . . . . It can no longer count on a steady stream of people coming in, on being the coveted institutional spot in the system," Ornstein said, pointing to the long list of governors who decided against running for the Senate this year.

While he said he believes that the Senate has many first-rate politicians, Ornstein said he finds that it "doesn't have the across-the-board strength" it had even five years ago.

"You've got an institution that's tried to develop far more expertise in policy detail than it can handle," he said. "They've got only 100 people, and the more detail they try to develop by spreading themselves thinner, the less detail they consume. They become more dependent on staff and they wind up at a competitive disavantage with the House."

The blame is spread widely, from the rules to the leadership to the members.

Some also cite a political climate in which party and institutional discipline have declined, replaced by a system of rewards and punishments that puts a premium on bucking the establishment, defying compromise and catering to outside interests.

In the reform-minded period of the 1970s, the Senate, like the House, moved toward greater democracy and decentralization. Meetings were opened, committee and subcommittee assignments were expanded and staffs were enlarged.

But, with its long tradition as the last outpost for expression of minority views, the Senate was tilted against efficiency, and the changes only accentuated the tilt.

Meanwhile, a new breed of politician appeared, reared on television and skilled in its use to establish a power base outside the Senate. With fewer hierarchical restraints, the newcomers became active players from the start, before the television cameras as well as before their colleagues.

Baker called it the "feisty freshman" phenomenon.

"When I first came here, it took me a long while to work up to my maiden speech, and I did so with great deference to the institution . . . . But now members hit the ground running, and that's all right," he said.

"Now, if no one listens, they don't care because they go outside and someone will listen to them with a TV camera. Freshmen are now driving the agenda of the Senate. It's not an unmixed blessing. It brings dynamic freshness into the system but it also makes it difficult to lead."

Even some of the freshmen seem to have mixed feelings.

In an age of instant and easy communications, it's easy to grandstand on just about any issue that comes along, and now there is no punishment for doing so, said Rudman, a freshman. "If I'd have tried to go up to the TV gallery as a freshman senator from New Hampshire 30 years ago," he added, "I'd have ended up with an office in Silver Spring."

Baker, who fought unsuccessfully to win Senate approval for televising floor sessions, contends that cameras at committee sessions, and their absence from the chamber, have helped shift senators' attention from deliberations of the Senate as a whole.

"If you'd televise the Senate, you'd get rid of a lot of the nonsense, and it would shift the center of gravity away from the committees to the floor, which is where it used to be, where it should be," he said.

With the collapse of other disciplines and the increased cost of campaigning, members also became more attuned to special interests and separate constituencies.

"Party discipline doesn't matter because parties don't matter," said Sen. David F. Durenberger (R-Minn.). "There's no discipline, just 30,000 special interests that we're all serving one way or the other."

In this context, the rules become something to manipulate, not follow.

The filibuster is a case in point. Unlimited debate, or the threat of it, is used almost routinely, as is cloture, by which debate can be limited by a vote of 60 senators.

At one point this fall, Quayle observed in a floor speech that the Senate had invoked cloture five times in the preceding 2 1/2 weeks on everything from highways to banking legislation, with three more cloture motions pending and others threatened.

By contrast, he said, it was invoked five times in the seven-year period after World War I and then only on such issues as the Treaty of Versailles and the World Court. Then, debate went on for months before cloture was approved; now it is customary to seek it the day a debate begins, he added.

"The Senate," he said, "has cloturitis."

A few years ago, the Senate thought it had solved the filibuster problem when it reduced the number of votes required for cloture from two-thirds of those voting to a flat three-fifths of the elected membership, or 60. This made it easier to get cloture, but prompted ingenious pursuit of other devices for delay by several Senate experts.

Now there are "post-cloture filibusters" that can delay action interminably, even after debate theoretically has been shut off; some senators stack up hundreds of amendments to fill in the time after their talking time has run out.

There also is a kind of pre-cloture filibuster in which senators talk at length about the routine motion to take up the bill.

Quayle and others say they hope these problems can be rectified by rules changes in January, but others question whether loopholes as yet unimagined may be found to get around these changes.

Ornstein also speaks of the proliferation of subcommittees and members' committee and subcommittee assignments. He participated in an effort to reform that system a decade ago, only to find that the problem reemerged as members sought waivers.

"Members complain they're spread too thin, but they don't want to give up any of their assignments," he said. "They want to give up their colleagues' assignments."

There was some grumbling -- especially as the Senate met for days without doing anything toward the end of this year's session -- that the leadership, especially Baker, was too reluctant to crack the whip.

But Ornstein said that even a consummate arm-twister such as Lyndon B. Johnson wouldn't have succeeded in today's Senate. "Just wait till a leader tells them to do something they don't want to do and see how they react," he said.

Said Baker: "If Lyndon were here today and tried to crack the whip, a member would simply go out on the Capitol steps before the TV cameras and raise hell . . . . That man would be a hero."

As Ornstein sees it, the leadership may change, the rules may change, but the Senate won't change until the members change. "The bottom line is that you've got to change the attitude of senators. It's not just an institution having an identity crisis, but it's 100 individuals not knowing exactly what their function is."