Senate Republicans, in a show of strength by conservatives, voted 22 to 21 yesterday to oust John H. Chafee (R.I.) as their third-ranking leader and replace him with the more conservative Thad Cochran (Miss.).

Chafee, the most liberal member of the Senate Republican leadership, was defeated by Cochran on a secret ballot as senators of both parties reelected their top leaders and filled out the rest of their leadership lineups for the 102nd Congress, which convenes in January.

Chafee, whom some conservatives faulted for splitting with the White House on such key issues as the recently vetoed civil rights bill, has held the No. 3 post as chairman of the Republican Conference for six years. Cochran was secretary of the conference for the same period.

Meanwhile, Senate Democrats unanimously chose Wendell H. Ford (Ky.) to succeed Alan Cranston (Calif.) as assistant majority leader, or party whip, and Charles S. Robb (Va.) was tapped by Majority Leader George J. Mitchell (Maine) as chairman of the Democratic Senatorial Campaign Committee.

Ford won the No. 2 Democratic leadership post without a contest after Cranston, one of five senators facing ethics hearings that begin Thursday on their involvement with savings and loan executive Charles H. Keating Jr., disclosed that he has prostate cancer and would not run again for the whip post.

David Pryor (Ark.) briefly explored the possibility of challenging Ford but stepped aside when it became clear that the Kentuckian had the votes.

Both Mitchell and the Senate's top two Republicans, Minority Leader Robert J. Dole (Kan.) and Assistant Minority Leader Alan K. Simpson (Wyo.), were unopposed for reelection.

In the other races for GOP posts, Phil Gramm (Tex.) defeated Mitch McConnell (Ky.) 26 to 17 for the chairmanship of the National Republican Senatorial Committee.

Sen. Don Nickles (Okla.) edged out Sen. Pete V. Domenici (N.M.) for the chairmanship of the Republican Policy Committee by 23 to 20 after an earlier tie in the contest. Sen. Robert W. Kasten (Wis.) defeated Sen. Christopher S. Bond (Mo.), 26 to 17, for the job of secretary of the Republican Conference.

In most, if not all, of the lower-echelon races, the winner was regarded as the more conservative of the two candidates.

In addition to Mitchell, Ford and Robb, the Democrats reelected Pryor as secretary of the Democratic Conference, the third-ranking leadership post, and retained Alan J. Dixon (Ill.) as deputy whip.

Daniel K. Inouye (Hawaii) will continue as chairman of the Democratic Steering Committee and Thomas A. Daschle (S.D.) as co-chairman of the Democratic Policy Committee with Mitchell.

Mitchell also chose his longtime chief aide, Martha Pope, to succeed Henry K. Giugni as Senate sergeant-at-arms, the chamber's second-ranking administrative post, sources said. Giugni announced last week that he is retiring to join a Washington lobbying firm.