Rep. Charles W. Stenholm (D-Tex.), the leader of a group of conservative southern Democrats, says he plans to challenge Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) when House Democrats elect their leader early next month.

Stenholm said yesterday that O'Neill is "further to the left than mainstream Americans" and that replacing him would be the first step in "putting the best face on our party to get more competitive in national elections. The Democratic Party has lost middle America."

Stenholm, 46, a west Texas cotton grower recently reelected to a fourth House term, said he realizes that "the odds are very much against me." He said he would forgo the challenge only if a more attractive candidate emerged in the Dec. 3 Democratic caucus.

Stenholm, who heads the 33-member Conservative Democratic Forum, was among the conservative Democrats who voted for President Reagan's tax and budget proposals in 1981 and 1982.

O'Neill, who became speaker in 1976, has been a leading liberal critic of the president on such issues as Social Security and programs for the poor, and has made clear that he will continue despite Reagan's landslide reelection. O'Neill, who will be 72 on Dec. 9, has announced plans to retire after one more two-year House term.

O'Neill told the Associated Press that he believes he has enough support to retain his post but that "anybody has a right to run."