A year and a half ago, when "New Right" Republicans packed the Senate's back benches and men such as Strom Thurmond (R-S.C.), Orrin G. Hatch (R-Utah) and Jesse Helms (R-N.C.) took over important committee chairmanships, the Senate looked as if it were changing into a citadel of super-conservatism.

But things haven't turned out that way.

"When you're trying to change the status quo and the status quo is entrenched, it's difficult," said Sen. John P. East (R-N.C.), one of the New Right freshmen and a Helms protege. "I don't think anyone had any illusions that it would be easy."

"Reality had a way of intruding," said Sen. Charles McC. Mathias Jr. (Md.), a Republican moderate.

"They were preoccupied with purifying the doctrine," said Sen. Mark O. Hatfield (R-Ore.), another moderate.

As the 97th Congress nears an end, it is not the New Right of the Republican Party but rather its Old Center, guided more by pragmatism than ideology, that calls the shots in the Senate.

Moderate conservatives like Majority Leader Howard H. Baker Jr. (R-Tenn.), Finance Committee Chairman Robert J. Dole (R-Kan.) and Budget Committee Chairman Pete V. Domenici (R-N.M.) are the dominant force, often setting the agenda for Congress as a whole and sometimes bending the administration to its will.

Working in cooperation with the administration, if not always in synchronized lock-step, they have swept almost everything else aside to consider President Reagan's economic program. And they have succeeded in large part, sometimes to the extent of forcing the administration to temper some of its more adventurous economic intiatives.

This had the effect, whether it was intended or not, of pushing such New Right issues as abortion and school prayer off to the side, where some of their post-election momentum of 1980 seems to have been lost.

Strong anti-busing legislation was passed by the Senate but only after a 10-month struggle that took its toll in members' patience, and the measure has virtually sunk out of sight in the House.

Abortion and prayer are now before the Senate, but they are mired in a parliamentary quagmire and chances of passage before adjournment in early October are uncertain.

The Family Protection Act, another favorite of the right, is languishing in a half-dozen or more Senate committees. The administration's proposal for tuition tax credits for children in private schools, also pushed by many conservatives, is in trouble. Capital punishment is unlikely to be taken up before year's end. Some of the right's more novel ideas, like denying crime victim aid to rape victims who have abortions as a result, have been quietly sidelined.

A look back at the post-1980 committee record of Thurmond, Hatch and Helms is instructive.

Thurmond may control the Judiciary Committee, but the major legislation from his committee that has passed Congress was an extension of the Voting Rights Act, strengthened even beyond its existing provisions.

Hatch may be chairman of the Labor and Human Resources Committee, but the balance of power on many issues rests with more liberal Republicans, Sens. Lowell C. Weicker Jr. (Conn.) and Robert T. Stafford (Vt.), who tend to vote with the Democrats in budget showdowns over spending for education and other domestic programs that Hatch would cut.

Helms heads the Agriculture Committee, where he was in a commanding position to attack food stamps. But Dole, pursuing damage control as chairman of Agriculture's nutrition subcommittee, was able to limit food-stamp funding cuts to far less than Helms wanted.

All three committee chairmen, principally Hatch, played a role in Senate passage of the balanced-budget constitutional amendment, another pet project of the conservatives. But, as an election-year antidote to the political poison of high deficits, it had support stretching far beyond the right. And the balanced-budget amendment, like the anti-busing measure, faces seemingly insurmountable problems in the House.

Perhaps even more illustrative of where real power in the Senate rests was the scene on the Senate floor late Thursday.

In the Senate of 18 months ago, voting against abortion would have seemed a far safer bet than voting to increase taxes. Yet Helms, who had to wait a year and a half to get his anti-abortion crusade onto the Senate floor, was fighting a filibuster against the measure for the fourth day when he had to stand aside for passage of the largely Dole-drafted tax increase bill. Helms voted against the tax bill and now faces resumption of the filibuster when Congress comes back to town after Labor Day.

By his tactics, including backing out of a time-agreement to limit debate on the anti-abortion measure, Helms has angered some of his colleagues to the point that they say personal frustrations could influence a crucial number of votes on the issue. "Some members are genuinely affronted," said one of his colleagues.

But Helms has always had more strength outside the Senate than in it, and he indicated on Friday that he has hardly given up. "We've got some troops in the field whispering sweet somethings in the ears of senators," said Helms, explaining that the grass-roots conservative lobby has marshaled its forces, phone banks and all, to lobby senators during the 2 1/2-week recess. "We pushed the red-alert button," he said.

Helms' clout outside the Senate is bolstered by his National Congressional Club, which was recently reported to have spent $8.7 million in the last 18 months, more than any of the other big political action committees.

The problem is that the New Right has inadvertantly activated other grass-roots forces, including lawyers and judges who are opposed to the way Helms would curtail pro-busing, pro-abortion and anti-prayer efforts: by limiting the courts' jurisdiction over these issues.

These quintessentially establishment groups thus find themselves on the same side of the lobbying fence as civil liberties and civil rights groups, which also report increased grass-roots support as a result of a perceived threat from the right.

"They've overreached themselves . . . they've caused the groups they oppose to get smart and employ their own successful tactics against them," said a Senate staff worker, noting the recent growth in groups like the American Civil Liberties Union. John Shattuck, director of the Washington office of the ACLU, reports that the ACLU's membership, now about 275,000, has grown by about 75,000 since 1980, more than at any other time.

There are other explanations for why the New Right didn't live up to its advance billing.

One, cited by Hatfield, was that its power was overblown from the start, by the press and others. "I'm not saying they have no influence but it was largely an image that had been falsely created," he said.

An even more frequently mentioned explanation is that senators like Helms have built a career upon causes, not experience in legislative craftsmanship, leaving them handicapped when they are thrust into positions of institutional leadership.

Moreover, some say, their ideological intensity makes compromise difficult and tends to lead to schisms, even within the ranks of the faithful, as happened when anti-abortion forces split over what specific legislation to support, delaying its consideration on the Senate floor.

In contrast, the Bakers and Doles of the Senate are experienced and adept at compromise. As legislators first and foremost, they also tend to be utility infielders. A Dole, for instance, can be juggling taxes, food stamps and voting rights at the same time.

By happenstance, or perhaps because of their longer tenure, the Republican centrists were also senior on the key fiscal committees, including Finance, Budget and Appropriations, when the Republicans took power early in 1981. And it was in these committees that the action of the 97th Congress has centered, giving them the opportunity to dominate center stage to the exclusion of other issues.

Thurmond is also an interesting case because, veteran legislator that he is, he has adjusted more easily to leadership than some of his conservative colleagues. "A real senator," said a liberal, in what passes in the Senate for unusually high tribute from a foe.

Some of the freshmen have quietly put some distance between themselves and the issues on which they were elected. A more senior senator, frustrated by the pressures of trying to accommodate ideological purity and legislative effectiveness, privately gripes about his "crazies."

But East and others make the case that the New Right has made progress, nonetheless.

"What we have now is a legislative climate in which these issues can be considered," he said last week. "Things are slow and long-term in a legislative process."