Senate Minority Leader Robert C. Byrd (D-W.Va.) brushed off a challenge by Sen. Lawton Chiles (D-Fla.) and claimed victory in advance yesterday as Senate Democrats prepared to choose their floor leader for the next two years.

But Byrd critics said Chiles still could win, contending that dissatisfaction with Byrd's leadership runs so wide and deep that Chiles, despite his late entry into the race and other problems, could well pull off an upset victory in the secret-ballot election today by the Senate's 47 Democrats.

Dismissing such talk, Byrd staged a "photo opportunity" for television cameras yesterday in which he talked on his office telephone to Democratic whip Alan Cranston (D-Calif.) and suggested that Chiles seemed to be running just to get something "out of his system."

Later, he told reporters he was "confident and sure, . . . no question about it" and added, "I'd say I could very well go home and spend the rest of the day with my wife but I don't do that. I work right up to the end."

Whatever the outcome, the race was indicative of strains within the Democratic Party as it reassesses its direction, image and leaders in the wake of President Reagan's landslide reelection last month.

Also, never before since Senate Democrats began electing their floor leaders in the early 1900s has there been a challenge to their incumbent party leader there, according to the Senate historian's office.

This, plus the fact that Chiles announced his candidacy only a week ago, long after Byrd got commitments of support from more than half the Senate's 47 Democrats, helps make Chiles the underdog.

So does the fact that some of Byrd's strongest critics would have preferred someone other than the moderately conservative Chiles. Some say they fear that the younger and more popular Chiles might be harder to dislodge than Byrd in two years, when the Democrats hope to reclaim control of the Senate.

Byrd's advantage from early commitments was underscored yesterday when it became known that Sen. Edward M. Kennedy (D-Mass.), whom Byrd unseated as majority whip in 1971, would give a seconding speech for Byrd's nomination today. A Kennedy aide said Kennedy had agreed to make the speech before Congress adjourned in October.

Moreover, in a not-so-subtle rejoinder to Chiles' claim that he offers a "new face" in the Democratic leadership, Byrd will be nominated by his new colleague from West Virginia, Sen.-elect John D. Rockefeller IV, who will be one of four authentic "new faces" among Senate Democrats next year.

Uncertainty remains for several reasons, however.

One is the fact that strange things happen in secret-ballot elections among senators.

For instance, Byrd's election as whip came with Kennedy contending right up to the end that he had enough commitments to win. Moreover, in the Senate Republican leadership race last month, at least two of the contenders for majority leader found that they had far more advance commitments than actual votes.

Another is widespread unease among Senate Democrats over Byrd's leadership, which is stronger on procedural skills than it is on articulation of policy, especially alternatives to Reagan administration policies. Some fear that a continuation of the present leadership could impede the party's chances of taking over the Senate in 1986.

There is also concern over the image of the aging party leadership in both houses, including House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.), 72, as well as Byrd, 67. The election last month of a feisty and articulate Republican leadership team headed by Sen. Robert J. Dole (R-Kan.) added to this concern.

Chiles, 54, ranking Democrat on the Budget Committee, has tried to address these concerns in several ways. He has projected himself as a force for change within the party, as a centrist figure who would signal the party's identification with "middle America" and as someone with a record of achieving bipartisan support for alternatives to Reagan programs.

Chiles, however, avoided victory claims and shied away from identifying key supporters, including those who would nominate him today.

One of his key behind-the-scenes supporters was Sen. J. Bennett Johnston (D-La.), who yesterday decided against challenging Sen. George J. Mitchell (D-Maine) for chairmanship of the Democratic Senatorial Campaign Committee. Other Democratic leaders appear to have no opposition, including Cranston and Sen. Daniel K. Inouye (D-Hawaii), the caucus chairman.