Sen. Harry F. Byrd Jr., the only independent in the U.S. Senate and bearer of the most famous name in contemporary Virginia politics, announced today that he will not seek reelection next year to a fourth term.

Byrd, who was named to his father's Senate seat in 1965, said he decided to leave Congress because he believes "that 18 years is long enough."

His announcement, which surprised many Democratic and Republican politicians here, signaled the end to a conservative coalition which took its marching orders and its view of limited government from two generations of Byrds. It also set the stage for a mad scramble for the Senate seat which Byrd and his father have held since 1933.

While no one entered the race today, Rep. Paul S. Trible Jr., a Tidewater Republican, who has waged a year-long campaign for the Senate, is expected to have the inside track for the GOP nomination. But Gov. John N. Dalton, who leaves office in January, is likely to come under intense pressure from party leaders and Byrd allies to challenge Trible.

Likely Democratic contenders in what is expected to be a wide-open free-for-all include former state attorney general Andrew P. Miller of Alexandria, the party's unsuccessful Senate candidate in 1978; former senator William B. Spong of Williamsburg and State Senate Majority Leader Hunter B. Andrews of Hampton.

Republican leaders, including President Reagan, have been wooing Byrd to join the Republican Party and make the race next year, but Byrd said today that as early as the summer of 1980 he and his wife, Gretchen, had agreed "that I would not seek another term." As the time for his announcement drew near, Byrd said he began "to have second thoughts" and it was not until last weekend that he confirmed his decision.

Byrd said today that results of the state's Nov. 3 election, in which Democrats swept the top three state offices, had "no influence in my decision." Neither, he said, did a statement by GOP state chairman Alfred C. Cramer that Byrd should either join the party or be prepared to face a strong challenge for reelection.

Byrd, who left the Democratic Party in 1970, undoubtedly will be urged by friends to reconsider his decision, but at today's news conference at the State Capitol, he indicated his decision is firm. Asked if he would accept a draft for the nomination from the Republican Party, he said: "I don't think that's at all likely to occur."

He noted that when his term ends in January 1983 he will have served in the Senate longer than all but six of the 47 Virginians who have been sent there in nearly 200 years.

The white-haired Byrd, who will be 67 in two weeks, was appointed to the Senate by his longtime friend, Gov. Albertis S. Harrison, on Nov. 12, 1965, upon the resignation of his father, Sen. Harry Flood Byrd. The elder Byrd had run the state's politics for decades through a courthouse-based organization known by its detractors as "The Byrd Machine" and by its admirers as "the organization." The senior Byrd died shortly after his resignation from the Senate where he was a major power.

The younger Byrd wielded much less power on Capitol Hill and early in his Senate career acknowledged that he lived in his father's shadow. Nonetheless "Little Harry," as he was known, was elected as a Democrat to a four-year term in 1966. In 1970 he became the second person in history to win a Senate seat as an independent and in 1976 won a third term by the largest vote ever given to a Virginia candidate.

Byrd said today he will "make no recommendations" about a successor, but hinted he might do so after the parties have picked their nominees. "There is no candidate I am pushing or have in mind," he said, adding that who takes his place, "is for the people of Virginia to decide."

Many politicans said Trible's activism may have been a factor in convincing Byrd to retire. While many Republicans were urging Trible to announce his candidacy regardless of what Byrd did, one Capitol Hill aide said "Trible played it smart -- he held off and now he can make it look as if he forced Byrd out."

The biggest challenge to Trible, 34, may come from Dalton, who had announced a year ago he would not seek the Senate seat. The governor, who cannot succeed himself, is already being urged to run for the Senate by some of the GOP's chief money men, who fear a possible repeat of the defeat suffered by J. Marshall Coleman in the race for governor.

"This is a crucial time for the Republican Party," said GOP fundraiser J. Smith Ferebee, a wealthy Richmond investor, who said he planned a personal appeal to Dalton. "We need to run the strongest candidate we can."

Dalton, who friends said was taken by surprise by Byrd's announcement, issued a brief statement praising the senator. Press Secretary Charles J. Davis said Dalton had "no interest" in running for the seat and had made "no change" in his plans to join a large Richmond law firm in January.

One party elder, suggesting Dalton might change his mind, said that while he understood Dalton's desire "to turn his sights to cash income," the governor would soon learn that "you get to be pretty obscure pretty fast" once out of office.

Dalton was not the only one surprised today by Byrd. Some of the senator's principal backers said they had given little thought to finding a suitable candidate, although they predicted a search would begin immediately. "I'm afraid some people were sitting on their fat-dumb-and-happy thinking Sen. Byrd would run again," said Richmond stockbroker Henry L. Valentine, another GOP fund--raiser.

One Republican on Capitol Hill said Byrd's decision produced "a great sigh of relief" among Virginia's nine-member GOP House delegation, many of whom feared a three-way race among the independent Byrd, a Democrat and a Republican would hurt their own chances for reelection.

Rep. Stanford E. Parris of Fairfax said Byrd's announcement "clarifies the political situation for all of us." Parris, who is likely to be challenged next year by former Democratic Rep. Herbert E. Harris II said Byrd's running as an Independent "would almost guarantee the election of a traditional, liberal-oriented Democrat."

Byrd discounted today any suggestion that reports that he had slipped in popularity or might be opposed by nominees from both parties played any part in his decision.

As far as retirement signaling the end of a political era, Byrd said: "I was under the impression that the Byrd machine ceased to exist some 16 years ago (at his father's retirement), and certainly it ceased to exist after 1970, when I ran as an independent."

After leaving the Democratic party, Byrd continued to vote with the Democrats for organizational purposes, which until this year's Republican takeover meant that he was part of the majority in the Senate. As a result, he held high ranking positions on the Finance and Armed Services committees.

Asked if he hoped that one of his two sons might some day also serve in the Senate, Byrd said: "I hope one of them would be active and seek public office, although I am very reluctant to encourage them because it is a tough life and a tough job."

He then invited his sons to "speak for themselves." Harry F. Byrd III, an apple farmer in Winchester who has shown no public interest in politics, strolled to the microphone grinning and said, "Not at this time." Thomas T. Byrd, the younger son who has joined the Republican Party and who for a time was reportedly considering seeking the GOP nomination for lieutenant governor earlier this year, echoed his brother. "I have no intention at this time," he said.

The senator reminisced about his years on Capitol Hill and said that while some aspects of the federal government have worsened since his arrival, "with the election of Ronald Reagan, the atmosphere in Washington has improved."

Byrd said he has "many wonderful friends in Virginia who wouldn't be caught dead in the Senate," while many of his congressional colleagues "want to stay until they are dead. I take a more moderate view," he said.