Rep. Charles W. Whalen Jr., a six-term liberal Republican from Dayton, Ohio, who recently was reported to be considering switching to the Democratic Party, announced yesterday he will not seek re-election to Congress.

Whalen said he was retiring because he believes 12 years should be the limit for service in Congress.

"The founders of our republic envisioned a Congress occupied by citizen-legislators, not political careerists," Whalen said in announcing his retirement.

He said he has no immediate plans except to "spend more time smelling the roses."

Recently House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) announced that three Republican congressmen had made overtures about switching to the Democratic Party, and there was widespread speculation that Whalen was one of them.

Although Whalen had always won reelection to Congress with about 70 per cent of the vote, he faced a county Republican organization that was hostile and far to the right of him on many issues. In 1976, Whalen for the first time had primary opposition, and his two opponents together got about 47 per cent of the vote.

While Whalen denied that he was one of the three unidentified Republicans on O'Neill's list, he acknowledged he had explored the option of switching or running as an independent. However, an Ohio law requiring that person be a member of a party for five years before he could become a candidate for office, prevented Whalen from running as a Democrat, and he ruled out running as an independent.

Whalen has been considered one of the more liberal Republicans in the House. When he served on the House Armed Services Committee he was known as one of the "fearless five" congressmen who angered then-Chairman Mendel Rivers by demanding scrutiny of military spending requests.

Whalen was an early proponent of an all volunteer Army and of a negative income tax welfare revision system.

Whalen is a member of the International Relations and District of Columbia committees.