Rep. Michael J. Harrington, a 41-year-old maverick Democrat from Massachusetts, announced yesterday he would not seek reelection.

The announcement was in a style consistent with his nine-year career - he blasted the congressional establishment.

Harrington conceded that a close race in 1976 and the prospect of a hard primary fight this year with two opponents helped him "make a decision rather than have one made for me." He said polls show he would have won the primary, but "running for the seat again, simply to prove I could hold it, was not a very compel-notion."

Harrington said the need to make money to but his five children through college was a compelling reason not to run. But mostly he criticized the current Congress in a way that he admits causes family friend Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.), to look on him with "bemusement and resignation."

Harrington said Congress was caught in a "tyranny of mediocrity" where members are more concerned about keeping up their "errand boy" role than about the kind of new ideas that might provide leadership for the country. He added that the House needed a "major institutional restructuring rather than the tinkering" done through recent "reforms." He said liberals were devoid of new ideas, reduced to "warmed-over new dealism," and, though conservatives are willing to "wipe clean the slate of old notions," he can't support "their priorities or their programs."

For Harrington, it was right in character. Elected as a devout anti-Vietnam war candidate in a special election in 1969, Harrington in his first speech on the House floor, attacked the seniority system. Since then he has regularly taken on the Pentagon and the Central Intelligence Agency, and is credited with forcing revelation of CIA involvement in overthrowing Chile's Marxist government with a bloody coup in 1972.

Harrington also forced the junking of a House committee set up to investigate the CIA, when he charged that its chairman was too close to the agency. Both Harrington and the chairman, Lucien Nedzi (D-Mich.), were taken off the committee and a new panel was formed.

Harrington's reputation as a maverick got him in political trouble. Some in Congress wanted to censure him for his role in revealing the Chile coup. And in 1976 Harrington was almost defeated by an unknown airline pilot. Bill Bronson, who charged that Harrington was neglecting his district.

While Harrington has always been a dissident, his stated disillusionment with Congress is becoming common. In fact, Harrington's retirement sets a new record for voluntary retirements from the Hill. The old record was the 56 members who retired in 1976, including 48 House members and eight senators.

With Harrington, the number of House members quitting after this session becomes 49. Eight senators have announced their retirement, for a total of 57.

The number has been steadily increasing since 1972, the first year that members who retired voluntarily could collect a generous pension. But higher pensions and scandals are not the only reason for the retirements anymore and the number of younger people retiring is increasing.

Increasingly, members cite frustration with the job - demands of constituents, too many votes, too much traveling back and forth, too many committee assignments, too much frantic activity with no time for thought or consideration of issues.

This year a new irritant has been added. Last year Congress voted an ethics code that puts a limit of $8,625 on the amount of outside income members can earn. Harrington deplored the limit and the kind of people who are coming to Congress and will come as a result of it. "I'd rather have dimensional villains than sterile managers," he said.

Harrington said he would like to continue to work on Northeast industrial problems after his retirement. But he didn't rule out returning to public service. Despite his criticism, he said, on the whole he enjoyed his experience in public life.