A three-way contest among Sens. Daniel K. Inouye (Hawaii), George J. Mitchell (Maine) and J. Bennett Johnston (La.) is shaping up for the leadership of Senate Democrats amid mounting indications that Majority Leader Robert C. Byrd (W.Va.) will step down from the post after next November's elections.

But maneuvering among Senate Republicans to succeed Minority Leader Robert J. Dole (Kan.) as he moves more actively into his campaign for the 1988 Republican presidential nomination has stalled because Dole has decided to hold on to the leadership post.

Dole "has absolutely no plans to step down" even as the presidential nominating contest moves into its most intensive and critical phase early next year, said a GOP leadership aide, noting that Dole has maintained a 95 percent voting record despite heavy campaigning this year.

Byrd, who has served as elected leader of Democrats in the Senate since 1977, said in an interview yesterday that he is keeping his "options open" for another bid for the leadership post and remains confident he would be reelected if chooses to run.

He said he has not had time to consider his long-range plans because of the pressure of pre-adjournment business in the Senate and will not announce a decision before early next year.

But several senators said they have heard directly or indirectly from Senate leaders that Byrd plans to relinquish the job in favor of two other powerful and prestigious posts to which he would be entitled by seniority.

With the retirement next year of Sens. John C. Stennis (D-Miss.) and William Proxmire (D-Wis.), Byrd will become the Senate's senior Democrat if, as expected, he wins a sixth term in the general elections. This would put him in line to succeed Stennis as president pro tempore of the Senate and chairman of the Appropriations Committee if Democrats retain control of the Senate next year.

In the peculiarly personal and discreet campaigning style of Senate leadership races, none of Byrd's would-be successors has announced his candidacy; none has indicated he would challenge Byrd if he were to run again for majority leader.

But, with varying degrees of directness, Inouye, Mitchell and Johnston have indicated interest in running if Byrd does not, and some senators said they have been contacted by one or more of them in what appear to be protective moves against preemptive strikes by possible rivals.

"I know what's going on. Senators are taking out a little insurance . . . and I would do the same," said Byrd, expressing no dismay at the tactics.

In previous years, Byrd has moved early to lock up support, sometimes a year in advance of the biennial leadership elections, which occur within a month or two of senatorial elections. In the case of earlier challenges to Byrd, this has had the effect of shutting off opposition before it could develop.

But Byrd has not done so this year, feeding speculation that he would not run for majority leader again. The fact that Inouye has begun to court his colleagues has been interpreted as another sign, because of Inouye's closeness to Byrd; many senators doubt that Inouye would make such a move unless he was confident that Byrd was preparing to step down.

As secretary of the Senate Democratic Caucus, Inouye is currently the third-ranking Democratic leader behind Byrd and Assistant Majority Leader Alan Cranston (Calif.), whom many senators expect to seek reelection to the No. 2 post. Last year there were unconfirmed reports of an understanding between Byrd and Inouye that Byrd would serve only one more term, clearing the way for Inouye to succeed him in 1989.

But Mitchell's star rose dramatically for a number of reasons, particularly his role as chairman of the Democratic Senatorial Campaign Committee during last year's elections, when the Democrats recaptured control of the Senate.

Although Johnston failed to muster enough support to mount a challenge to Byrd two years ago, he also has earned praise for handling of major legislation on the Senate floor, including the omnibus appropriations bill that passed the Senate with relative ease last week.

On the Republican side, Sens. Pete V. Domenici (N.M.) and Ted Stevens (Alaska) have expressed interest in succeeding Dole.