Independent Virginia Sen. Harry F. Byrd Jr. broke a family tradition of two generations today and endorsed a presidential candidate, denouncing President Carter's "dismal" record and saying he will vote for GOP nominee Ronald Reagan.

The move is likely to help the former California governor pick up support among the state's large and crucial segment of undecided voters.

The surprise endorsement by Byrd, a former Democrat who has won reelection twice in the last 10 years as an independent, is also likely to enhance the political prospects of his Republican son and is seen by some observers here as a signal that Byrd intends to retire from his Senate seat in 1982.

The silver-haired conservative, whose family has represented Virginia in the U.S. Senate since 1933, journeyed to the state Capitol for his first press conference here in at least three years to announce his decision and to blast the Carter administration.

"His policies have no consistency," Byrd said of Carter. "His twists and turns are such that one cannot determine if he is going East or coming back. His credibility and, indeed, his reliability, have justifiably been subject to question."

Two recent Virginia polls have shown Reagan holding a slim lead in the one southern state that Carter lost to Gerald Ford four years ago. Both polls suggested the key to victory here, as in much of the country, lies with undecided voters who the polls indicate make up nearly 20 percent of the state's electorate.

Republican officials hailed Byrd's endorsement today, saying it could be a major factor in swaying the undecideds.

"Any time you can have someone whose last name is Byrd say something nice about a Republican candidate, it's got to be helpful," said John Anderson, Reagan's state campaign chairman.

"This is a very positive move, nothing but a plus factor," said Robert Hausenfluck, executive director of the Reagan-Bush Committee of Virginia. "It will have a magnificent effect on the independent voters."

Byrd said it was the high level of undecided voters that led him to break with the tradition of "golden silence" first established a generation ago by his father, the late U.S. Sen. Harry F. Byrd Sr.The senior Byrd was founder and principal cog in the Byrd Machine, the Democratic organization that ruled this state for nearly 40 years.

To the anger of many Democrats the senior Byrd developed a policy of refusing to endorse Democratic presidential nominees (starting with Adlai Stevenson in 1952), freeing his followers to work for the Republican national ticket. Partly as a result, Republicans have carried Virginia in six of the last seven presidential campaigns, with only Lyndon Johnson bucking the trend.

Byrd, who said with a smile that he could not recall if he has ever before formally endorsed a presidenial nominee, predicted that Reagan would carry the state, but only if "the large majority of the Virginia people (who) are of a conservative philosophy" go to the polls next month.

Byrd's endorsement will be seen by some as a political prize captured for the GOP by his Republican son, Winchester newspaper publisher Thomas T. Byrd, who is considering running for lieutenant governor next year. The younger Byrd, who silently watched his father announce the endorsement today, said he played no role in the decision.

Sen. Byrd denied that his son's policital future as a Republican helped influence today's announcement. Byrd also said he had not made up his mind on whether to seek another Senate term in 1982.

But several political insiders said they interpreted Byrd's endorsement of Reagan as a signal that he planned to retire. They said the endorsement would further weaken Byrd's already tenuous status as a member of the Senate's Democratic caucus and could jeopardize his seniority.

"I just can't see him staying in the caucus now that he's turned his back on their nominee," said one high-ranking state Republican. The office of Majority Leader Robert C. Byrd of West Virginia, the Senate's chief Democrat and no relation to Harry Byrd, said he would have no comment today on Harry Byrd's action.

Harry Byrd himself said he did not expect his endorsement to lead to his expulsion from the caucus. "This matter of the election of a president is totally unrelated to any other matter," said Byrd.

Carter supporters said they were disappointed but not surprised by the Byrd endorsement, which they contended would not have any real impact on voters since most already knew where Byrd's conservative heart lay.

"He (Byrd) has a lot of personal popularity in this state, but I haven't met a single undecided voter who told me he was waiting to hear what Sen. Byrd had to say," said George Gilliam, Carter's Virginia campaign chairman.

Byrd made no mention in his prepared statement of his recent feud with Carter over the president's nomination of a black state judge to the federal bench, an appointment that Byrd has vehemently opposed and helped keep bottled up in the Senate Judiciary Committee.

But he said afterward that Carter's handling of the nomination, in which Byrd argued that the president ignored a pledge to pick nominees only from a list submitted by Byrd, was one indication of Carter's lack of credibility.