An effort to topple Senate Minority Leader Robert C. Byrd (D-W.Va.) fizzled yesterday as Democrats voted, 36 to 11, to reject a challenge by Sen. Lawton Chiles (D-Fla.).

Despite widespread expressions of dissatisfaction with Byrd's leadership, Chiles was unable to rally his colleagues for a variety of reasons, including the fact that Byrd obtained commitments from a majority of them long before Chiles' late entry into the race a week ago.

Some senators also expressed misgivings about trading Byrd for the moderately conservative and low-key Chiles, preferring instead to wait until 1986 when the Democrats hope to recapture the Senate and elect the majority leader.

The election of Byrd to his fifth term as Democratic leader, along with the reelection of Sen. Alan Cranston (D-Calif.) as party whip and Sen. Daniel K. Inouye (D-Hawaii) as caucus chairman, was in stark contrast to the top-to-bottom leadership overhaul approved by Senate Republicans last week.

The only change was the selection of Sen. George J. Mitchell (D-Maine) as chairman of the Democratic Senatorial Campaign Committee to replace Sen. Lloyd Bentsen (D-Tex.), who did not seek another term as chairman.

Chiles' candidacy was prompted in part by fear that Majority Leader-elect Robert J. Dole (R-Kan.), a resourceful and articulate legislative leader, will put the Democrats at a crippling disadvantage as they try over the next two years to build a case for reclaiming control of the Senate.

Byrd is a skillful parliamentarian and is attentive to his colleagues' interests but has been faulted, nearly always privately, for projecting the image of a tactician and failing to articulate forceful policy positions.

"I'm not just a pretty face," Byrd said. "I will respond to the needs as I always have."

Once Byrd was reelected in the secret-ballot vote, Democrats rallied publicly behind him, saying he will be a strong leader.

Sen. J. Bennett Johnston (D-La.), a Chiles supporter, said Byrd won because he is "a centrist who brings opinions together . . . a master parliamentarian . . . hardworking, patient and accommodating to all senators."

An aide to another senator suggested other factors included Byrd's control over new committee assignments, which could affect about half the Democrats. "Watch those committee assignments," the aide said.

In statements after the vote, Byrd, 67, who had not been opposed from within the party since his election as majority whip almost 14 years ago, picked up some of Chiles' campaign themes but suggested there will not be "anything . . . new about Robert C. Byrd" in the future.

Echoing the 54-year-old Floridian, Byrd talked about responding to President Reagan with specific legislative alternatives, including bipartisan ventures with Senate Republicans, and targeting the Democratic Party to appeal to "Middle America."

Responding to criticism that he and House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) present an aging, tired image of Democratic leadership, he appeared to put some distance between himself and the more liberal O'Neill, 72.

"Our party in the Senate represents the look forward, it represents Middle America," Byrd said. He declined to say whether he thinks the House leadership represents the same thing.

Later Byrd issued a clarifying statement praising O'Neill's "hard work in his career as a successful speaker" and said he implied no criticism in his earlier remarks.

Byrd talked about change as something he always promoted. "You don't have to be under 30 to be able to deal with change, and we have worked diligently and painstakingly to develop a unified minority party . . . . We're ready to deal with the problems that face this country, and we're ready to work to regain control of the U.S. Senate," he said.

Chiles and several others said Byrd had stressed during the closed caucus a need for responsiveness to change. "Everything I heard sounded different today," Chiles told reporters afterward. "Who knows what he may do in two years," he added. "He may grow 10 feet tall."

Chiles asserted that his candidacy, despite his poor showing, may encourage others to challenge Byrd. Expressing no regrets about running, he said, "A challenge has been made now, so it's not so it can't be done," an apparent reference to the fact that his was the first challenge of an incumbent Democratic floor leader in Senate history.

But some other Democrats, including Chiles supporters, suggested that Byrd would continue to have a strong advantage for the leadership post, especially if the Democrats win control of the Senate in 1986.

Johnston, one of the few Chiles supporters who let his preference be known publicly, said he thought that there is a "very good chance" Byrd will be elected majority leader if the Democrats take control in two years.

Sen. Patrick J. Leahy (D-Vt.) agreed but added that a change in leadership is likely if the Democrats fail.

Byrd served three terms in the House and was first elected to the Senate in 1958. He unseated Sen. Edward M. Kennedy (D-Mass.) as party whip in 1971 and succeeded former senator Mike Mansfield (D-Mont.) as majority leader in 1977. He became minority leader in 1981 when the Republicans took control of the Senate.