Some left out of ambition, others out of disillusion. Some were ousted unceremoniously in the primaries. A few retired gracefully.

Fifty members of the House of Representatives and three senators are not running for reelection this year, including some of the most famous and a few infamous lawmakers on Capitol Hill.

Redistricting, that once-in-a-decade game of musical chairs, forced many House members into a choice between retirement or running against a fellow incumbent, as northeastern and midwestern states yielded seats to the fast-growing Sun Belt.

In the Senate, the retirement of Virginia's Harry F. Byrd Jr., Congress' only independent, marks the end of an era in Virginia politics, although the once-powerful Byrd machine long since released its grasp on the state's affairs. The courtly, silver-haired lawmaker had succeeded his father, who was elected to the Senate in 1933.

California's S.I. Hayakawa, 75, decided to retire rather than face stiff challenges from several younger, more popular Republicans in the primary.

The short, peppery linguistics professor, who became famous for standing up to student radicals as president of San Franciso State College, had gained a reputation in the Senate for tactless remarks and falling asleep at meetings.

New Jersey's Nicholas Brady, a Republican appointed as a caretaker after Democrat Harrison A. Williams Jr. resigned after his conviction in the Abscam affair, also will retire. Williams faced a vote of expulsion from the Senate after he was convicted of taking a bribe from FBI agents disguised as phony Arab sheiks in the Abscam "sting" operation.

In the House, 29 Democrats and 21 Republicans are leaving, 10 more than in 1980 but not an unusual number. In addition, two seats are vacant, due to the death of Rep. Adam Benjamin Jr.(D-Ind.) in September and the resignation of Rep. Frederick W. Richmond (D-N.Y.) in August after pleading guilty to charges of tax evasion and possession of marijuana.

Among the Democrats, several of the House's most senior and influential members are retiring, including Rules Committee Chairman Richard Bolling (Mo.), 66, a force behind the post-Watergate reforms, which among other things greatly weakened the old House seniority system. Bolling, who plans to teach government and write a book, never achieved his ambition of becoming speaker of the House, but behind the scenes he was one of the half-dozen most powerful members of the Democratic leadership.

Another veteran Democrat, New York's Shirley Chisholm, 57, admitting that she had come to find the job "tedious and frustrating," decided to leave. Chisholm, the first black woman ever to win a seat in the House 13 years ago, ran for president in 1972, but in recent years she had toned down the angry style that was her trademark.

Nonetheless, in a written statement announcing her retirement, Chisholm took a swipe at the Reagan administration, saying, "It has become increasingly difficult to carry the tragic messages back from Washington to the jobless, homeless and hopeless Brooklynites."

Other senior Democrats who chose not to run include two liberals, New York's Jonathan B. Bingham, 68, an influential member of the Foreign Affairs Committee, and Wisconsin's Henry S. Reuss, a former Banking Committee chairman, who has had heart trouble.

A number of "Boll-Weevil" conservative Democrats also retired, including North Carolina's L.H. Fountain, Georgia's Jack Brinkley, Texas' Richard White and Mississippi's David R. Bowen.

Bowen chose to retire rather than seek a sixth term in a district that was redrawn to one having a 54 percent black population.

Republicans are also losing senior members, including former minority leader John J. Rhodes (Ariz.), M. Caldwell Butler (Va.), an active member of the Judiciary Committee, and Clair W. Burgener, 60, secretary of the Republican Conference, who says he wants to teach political science, go into the construction business with his two sons, learn Spanish and take piano lessons.

Redistricting took its toll on Illinois Republicans, including Rep. Robert McClory, the ranking member of the Judiciary Committee, who chose to retire rather than run against fellow Illinois Rep. John Edward Porter when the Illinois delegation was reduced from 24 to 22 seats.

Twelve-term Illinois Republican Edward J. Derwinski, a senior member of the Foreign Affairs Committee, lost to his close friend, Rep. George M. O'Brien, with whom he often commuted to work.

Illinois Republican Tom Railsback, a moderate, was defeated by a conservative challenger when several rural areas were added to his district. Chicago Rep. John G. Fary, a Democrat and a stalwart of the late Mayor Richard Daley's machine, also was defeated.

While many of those who leave voluntarily are near retirement age, more and more younger members are choosing not to run because of the toll serving in Congress takes on their personal lives, or because they are disillusioned with the ideological compromises they must make to raise funds from political action committees.

"I find that I no longer have enthusiasm for my work," said Rep. William M. Brodhead, who was serving his first term as head of the Democratic Study Group, an influential group of liberals. He is returning to private law practice and plans to spend more time with his family.

Quest for higher office claimed a number of House members. Two Democrats, Reps. Anthony (Toby) Moffett (Conn.) and Floyd J. Fithian (Ind.), are running for the Senate, as are six Republicans: Reps. Millicent Fenwick (N.J.), Paul S. Trible Jr. (Va.), David F. Emery (Maine), Robin L. Beard (Tenn.), James M. Collins (Tex.) and Cleve Benedict (W.Va.).

Three House Republicans were defeated by San Diego Mayor Pete Wilson in the race for the California Republican Senate nomination: Barry M. Goldwater Jr., Robert K. Dornan and Paul N. (Pete) McCloskey Jr. James D. Santini (D-Nev.), was narrowly defeated in a bid for Sen. Howard W. Cannon's seat.

Three Democrats, Bo Ginn (Ga.), James J. Blanchard (Mich.) and Allen E. Ertel (Pa.), left to run for governor, as did two Republicans, L.A. (Skip) Bafalis (Fla.) and Clarence J. Brown (Ohio).

Among those defeated in primaries were Ronald M. Mottl (D-Ohio), who was attacked for his support of the Reagan program, and Indiana Rep. David W. Evans, who lost to another incumbent, Andrew Jacobs Jr., despite the fact that he knocked on 30,000 doors.

Two Pennsylvania Democrats, Don Bailey and Joseph F. Smith, also lost to other incumbents in races forced by redistricting. A third Pennsylvanian, Marc L. Marks, a Republican, chose to retire with a back problem after excoriating President Reagan with a speech calling for an end to his "murderous mandate."

Marks, a trial lawyer from Sharon, was more candid than most, as he announced his retirement. "I've had a very undistinguished career in the six years I've been here," he said. "When I came, I expected I could change the world, and I found out pretty damn quickly I couldn't."