Republican Mitch McConnell wasn't even sure the voters of Kentucky had elected him to the Senate before the courtship ritual began.

With varying degrees of boldness and modesty, all five candidates for Senate majority leader had called, wired, written, visited or campaigned for him before the Nov. 6 election, most of them sending their greetings on Election Day as well.

The contacts intensified after McConnell learned that he had defeated Sen. Walter D. Huddleston (D-Ky.). Within a few days, McConnell had received at least 50 calls from candidates for Senate leadership posts or their surrogates.

Thus was McConnell initiated into the tribe. He was participating in one of its most hallowed rites: the selection of a new chief.

In this case, the new chief will be the successor of Majority Leader Howard H. Baker Jr. (R-Tenn.), who has led the Senate since Republicans took control from Democrats in 1981 and is now retiring, possibly to seek the GOP presidential nomination in 1988.

The Senate's 53 Republicans will gather behind closed doors at the Capitol next Wednesday to choose their leader from among Sens. Robert J. Dole (Kan.), Pete V. Domenici (N.M.), Richard G. Lugar (Ind.), James A. McClure (Idaho) and Ted Stevens (Alaska). There is no clear front-runner.

Theirs is an unusual event in American politics.

It is at the pinnacle of politics, yet it is strangely apolitical. Sardonic observers have suggested it has elements of the process by which the College of Cardinals selects a pope as well as the less celestial aspects of a fraternity rush.

The competition is conducted on such a personal level, not visible to the public, that most normal tools of the political trade are abandoned by these consummate politicians.

No polls, no television blitzes, no fancy brochures. Even such qualities as ideology, regional balance and legislative prowess play a subsidiary role. The White House is politely told to keep hands off.

Most critical to the outcome, according to senators, are such basic human qualities as friendship, loyalty, trust, favors owed and favors expected. Sensitivity to creature comforts, such as work schedules, cannot be overestimated, they say. Also lurking in the shadows is the complex interplay of personal ambitions, including members' prospects for ascendancy on committees and other aspects of an individual senator's power and prestige.

"What it boils down to is who you feel most comfortable with," said Sen. Dan Quayle (R-Ind.), one of the few Republicans not campaigning for one of the six leadership posts at stake on Wednesday.

The unusual rules by which the GOP election is conducted leave even the top contenders in doubt about whether they might wash out early or survive to the end, an unsettling choice that could lead to some dropouts before the vote.

Balloting in the caucus continues until someone wins a majority. In the first and each successive vote, the candidate getting the fewest votes is eliminated, which gradually whittles the field to two.

Thus, it is possible that a candidate who is many senators' second choice -- and who might have won under other rules -- could be knocked out in the first or second round. It is also a perfect climate for sudden shifts, quick deals and alliances of convenience.

The fluidity of the situation is compounded by the fact that, as of mid-week, a third or more of the 53 Republicans had made no commitments in the majority leader's race, according to candidates or their aides. Some may hold out to the end.

A few senators are believed to have given dual, if not multiple, indications of support. Dole tells of one senator who introduced him at a gathering as a magnificent choice for majority leader, only to say later that he was committed to another contender.

Recent Senate history is replete with examples of senators who went into party caucuses holding enough commitments to win, only to come out empty-handed. Baker became floor leader in 1977 when one or more votes that had been promised to front-runner Robert Griffin (R-Mich.) came Baker's way in the secret ballot. Sen. Edward M. Kennedy (D-Mass.) similarly lost reelection as Democratic whip to Sen. Robert C. Byrd (D-W.Va.) in 1971.

Another imponderable is the apparent strategy of six party moderates to vote as a bloc, at least after the first vote, and thereby accentuate its influence on the outcome.

The "Gang of Six" -- Sens. Mark O. Hatfield (Ore.), Charles McC. Mathias Jr. (Md.), Robert T. Stafford (Vt.), John H. Chafee (R.I.), Lowell P. Weicker Jr. (Conn.) and Mark Andrews (N.D.) -- plans to meet the day before the caucus to determine their strategy. While few in number, their votes could be critical in tight balloting.

The selection of a new majority leader could have a major impact on relations between Congress and the White House in President Reagan's second term and could affect the ability of Senate Republicans to withstand an expected knockout bid by the Democrats to reclaim control of the chamber in 1986.

GOP Senators reportedly are looking for someone who will stand up to the White House if necessary, while not provoking unnecessary tensions that the Democrats might exploit.

Significantly, McClure is using his record as a conservative Reagan loyalist to make the point that he is best positioned to stand up for the Senate when it comes in conflict with the White House.

"In that sense, I am freest to be the Senate's man," he tells colleagues.

Dole and Domenici have proven records of talking back to the White House. That both reassures and worries their colleagues.

The Senate leadership race could also affect the lineup for the Republican presidential nomination in four years. A victory for Dole could improve his standing for an expected presidential bid in 1988, while the embarrassment of defeat could damage his chances before he gets to the starting gate.

This has given rise to rampant speculation among his colleagues that Dole may drop out of the race unless he's reasonably certain that he will win.

Dole, grumbling that the rumor was started by one of his rivals to undercut his chances, insisted last week that he has no intention of bowing out, although he allowed in passing that "something might happen, who knows?" to influence his plans.

These factors and others have contributed to a significant reshaping of the contours of the campaign since it began nearly a year ago after Baker announced his retirement and to a high degree of uncertainty over the likely outcome.

At the start, it was widely assumed that Dole, chairman of the Finance Committee and a virtuoso legislative performer, was the front-runner and that Stevens, who has made no great mark as party whip, was trailing the pack. It also was assumed that the less controversial Lugar, who was collecting chits as chairman of the National Republican Senatorial Committee, was a strong dark horse who could emerge as a compromise.

But many senators have come to fear that Dole's presidential ambitions could get in the way of his stewardship of the Senate. Others remember with some personal pain the lash of Dole's sharp tongue. A few also are concerned that Dole, if elected, would be succeeded as Finance Committee chairman by the relatively more liberal Sen. Bob Packwood (R-Ore.).

Lugar also has problems. Some of his most ardent supporters say privately that they fear he may be the odd man out on the first or second ballot, even though they contend he will be a strong contender at the end if he can survive that long. He entered the race late, and some say he has never caught fire. As much if not more than Dole, he has committee chairmanship complications.

As Dole and Lugar appeared to be faltering, Stevens rose as a force in the race, even to the point that Baker suggested recently that Stevens is "probably the front-runner."

Stevens has a legendary temper that concerns some colleagues, but he has tried to tame it, reinforcing his more amiable qualities, which include attentiveness to the personal needs of his colleagues. He has probably campaigned harder than any of the others, pitching his candidacy on a distinctly personal basis that fits the nature of the contest.

Perhaps more importantly, he does not knock over any dominoes on the leadership ladder of senatorial committee chairmanships, a point that he is not hesitant to make to colleagues with jobs at stake in the outcome. The reason is simple: Stevens does not hold a committee chairmanship and is not in line to get one.

Most of the others join Dole in having succession problems.

McClure would be succeeded as chairman of the Energy and Natural Resources Committee by Weicker, an eastern moderate with an environmental and consumer record that unsettles the energy industry.

A victory by Lugar would be likely to set off a chain reaction, which, among other things, could wind up making either conservative Sen. Jesse Helms (R-N.C.) or the more liberal Mathias head of the Foreign Relations Committee. Its current chairman, Sen. Charles H. Percy (R-Ill.), was defeated for reelection.

Helms reiterated last week that it is his "intent" to remain as chairman of the Agriculture Committee, but he did not close the door on taking the Foreign Relations chair, which he can claim under seniority rules if he chooses.

Lugar is behind Helms on Foreign Relations but would not be eligible for the chairmanship if elected majority leader. Behind Lugar is Mathias, whose support for arms control and other generally liberal positions would almost assuredly increase right-wing pressure for Helms to take the post.

Helms has indicated to colleagues that he is awaiting the outcome of the leadership race before giving the final word on his chairmanship decision.

Lugar was sufficiently concerned about the Helms-Mathias problem that Wednesday he sent out a letter urging his colleagues to ignore the committee chairmanship implications of his candidacy.

If he became majority leader, Domenici would be succeeded as chairman of the Budget Committee by conservative Sen. William L. Armstrong (R-Colo.). Domenici's main problem appears to be surviving the first cut.

McClure's problem may be the reverse. As the most conservative contender, he probably will go into the first round of voting with a substantial bloc of votes. But it is unclear whether he can expand it in future ballots.

Regardless of how the majority leader race turns out, the leadership contests as a whole are likely to tilt control of Republican Party machinery of the Senate toward the conservative Rocky Mountain region, reflecting an earlier trend in Senate Republican membership.

Among those running for other GOP posts are:

*Sens. Slade Gorton (Wash.), Robert W. Kasten Jr. (Wis.) and Alan K. Simpson (Wyo.) for deputy leader, or whip.

*Rhode Island's Chafee and Sen. Jake Garn (Utah) for Republican Conference chairman.

*Sens. Thad Cochran (Miss.) and possibly Rudy Boschwitz (Minn.) for conference secretary.

*Sens. John C. Danforth (Mo.) and Armstrong for Republican Policy Committee chairman.

*Sens. John Heinz (Pa.), Malcolm Wallop (Wyo.) and possibly Boschwitz for chairman of the National Republican Senatorial Committee, which runs the party's campaign machinery.