Conservative House Republicans failed yesterday to oust Rep. John B. Anderson of Illinois from the third-ranking leadership post but installed two of their own in lesser posts at a party caucus.

Anderson defeated conservative Rep. Tom Kindness (Ohio), but conservative Rep. Bud Shuster (Pa.) beat moderate Rep. Bill Frenzel (Minn.) for chairman of the policy committee and conservative Rep. Trent Lott (Miss.) defeated Reps. Lawrence Coughlin (Pa.) and Willis Gardison (Ohio) for chairman of the Republican research committee.

Other top leaders of both parties were reelected without opposition yesterday at the party organizing sessions for the House in the next Congress.

While Republicans, who have little to say about the rules governing the next Congress, finished their work yesterday, Democrats were just beginning to discuss some 52 rules changes on their four-day agenda.

In one major change adopted yesterday, Democrats extended to six years from four the time a member can serve on the House Budget Committee, the only one with a rotating membership. This almost assures the reelection of Rep. Robert Giamo (Conn.) as Budget Committee chairman in January.

Put off until today were rules changes that would automatically prevent a convicted member of Congress from serving as a chairman of a committee or a subcommittee or from voting in the House until he has served his sentence. The move would affect Rep. Charles Diggs (D-Mich.), who has been sentenced to three years in prison for taking salary kickbacks from his staff. Another member, Rep. Daniel Flood (D-Pa.), has been indicted on bribery charges.

In the contest for chairman of the conference of all House Republicans, Anderson was reelected over Kindness 87 to 55. It was the third time in Anderson's 10 years in the post that he has confronted a conservative challenge.

Anderson, 56 and a House member for 18 years, is widely regarded as one of his party's most articulate orators, and is giving some thought to running for president. But House conservatives consider him too far left of the GOP mainstream to be among the most visible spokesmen for House Republicans.

Kindness, a second-termer, was nominated by Rep. Gene Snyder (Ky.), who said Anderson supported the Republican Party position less than half the time, a statistic that Anderson's supporters deny. Snyder conceded that Anderson is a fine speaker: "I just wish he was on our side more."

Rep. Barber Conable (N.Y.) insisted Anderson had a good party support record and said the party should not want "docile or passive" leadership.

Minority whip Robert Michel (III.) said the party needed a "man of stature to harmonize" differing views.

Kindness said later he would have won except for the "unusual degree of participation by the incumbent leadership."

Shuster was elected chairman of the House Republican Policy Committee by a vote of 30 to 55 over Frenzel, who had been chairman of the Republican research committee and was trying to move up a notch. Shuster campaigned hard for support from the 36 freshman Republicans at their orientation sessions last week.

Lott, a six-year member and a prominent conservative on the House Rules Committee, was elected chairman of the Republican Research Committee with 82 votes, more than the combined total received by moderates Coughlin and Gradison.

The policy committee sets Republican positions on major issues, and the research committee gives the House Republicans background material and statistics to use in debate.

A rumored attempt by conservatives to block designation of liberal Rep. Silvio Conte (Mass.) as ranking minority member of the House Appropriations Committee did not materialize.

In the Democratic caucus, the Budget Committee membership extension to six years from four was offered by Rep. David Obey (Wis.), who argued that an "extremely tough year for the Budget Committee" lies ahead and it couldn't afford to lose its most experienced members. He said the rotation period meant the chairman barely learned his job before he had to give it up, and with an inexperienced chairman the House committee was at a disadvantage when it meets with the Senate committee, on which membership is permanent.The proposal passed, 50 to 30.

Obey also proposed allowing a member who is elected chairman during his last two-year period to serve another two years, for a total of eight. That change could benefit Obey, who was in line to be a candidate for chairman this year if Giaimo's term was not extended.

Rep. Abner Mikva (Ill.) argued against changing the rules for the sake of people now on the committee. Mikva said the proposal was "gradually wiping away rotation," which he said had been adopted to ensure fresh input on the committee. This Obey proposal was approved 61 to 30.

Today Rep. Peter H. Kostmayer (Pa.) is expected to offer proposals that would prevent a convicted member from chairing a committee or subcommittee or from voting on the floor until the sentence is served.

Diggs, who was chairman of the District of Columbia Committee, voluntarily agreed to give up that chair after his conviction, but he wanted to keep the chairmanship of an African subcommittee on the House International Relations Committee.

House rules state that a convicted member should refrain from participating in committee business or voting until he is reelected. Diggs was reelected in November, after his conviction. Kostmayer would drop the reelection provision from the rules. He said he would also like an automatic vote on expulsion from the House for a convicted member after appeals are exhausted.

"I don't think convicted felons should be allowed to vote in Congress. I don't think they should be even allowed to serve, but pending their appeals we're in a dilemma because we want to be fair," Kostmayer said.

A proposal by Rep. Matthew McHugh (N.Y.) would require a caucus vote on all subcommittee chairmen who have been convicted. censured, reprimanded or otherwise admonished by the House.

In addition to Diggs, Rep. Charles Wilson (Calif.), chairman of a post office subcommittee, would be affected. Wilson was reprimanded for accepting money from South Korean businessman Tongsun Park, key figure in the Korean influence-buying scandal.

Currently only appropriations subcommittee chairmen must be elected by the caucus. Flood, who is chairman of the Labor-HEW subcommittee of appropriations, intends to seek reelection despite his indictment.

Reelected without opposition were top leaders of both parties -- Speaker Thomas P. (Tip) O'Neill (D.-Mass.), Majority Leader Jim Wright (D.-Tex.), Majority Whip John Brademas (D-Ind.), Caucus Chairman Tom Foley (D-Wash.), Minority Leader John Rhodes (R-Ariz.) and Minority Whip Michel.