The 1.7 million federal retirees who would be put on a 4 percent COLA (cost-of-living adjustment) diet for the next three years come in all shapes and sizes, but have a few important things in common:

All either worked for Uncle Sam long enough to earn an annuity, or were married to somebody who did. All have seen promised benefit cutbacks in recent years.

Many feel that politicians of both parties--especially Republicans and particularly President Reagan--con them at election time with pension-protection promises that fade after the votes are counted.

Some retirees are doing well, thanks to the government retirement system. But the majority are living modestly. They watch the Consumer Price Index the way deep-sea fishermen watch the barometer.

Earlier this week the Senate Governmental Affairs Committee went along with the budget (already approved by the Senate and House) which caps the next three annual pay and pension adjustments for federal workers and retirees at 4 percent.

Retirees now get a raise each March to match the previous year's rise in living costs. Five months ago they got an 8.7 percent raise.

Lots of retirees think they have gotten a bad rap from the media, which has frequently quoted a few retired members of Congress and former federal officials who complain that their own retirement checks are embarrassingly high.

Some points about retirees:

* They are scattered across the nation (heavy in Florida, California, Virginia and Texas) and 110,000 live in foreign countries. There are 100,000 former military and feds here.

* Their average age is 66.9 years. Their average age at retirement is 59.3 years.

* There are 1.3 million retirees; 458,000 survivors.

* The average monthly annuity is just over $1,000. The average survivor benefit is just under $450.

* More than 1,000 get monthly pensions of less than $50. About 74,000 get less than $200 a month, and 270,000 get less than $500 a month.

* About 10,000 retirees (mostly long-service, high-salaried officials and members of Congress) get about $3,000 monthly.

* Percentage raises for retirees during the late 1970s period of high inflation went up faster than pay of federal employes, because pay was frequently capped by Congress or the White House.

* Some top-paid workers who retired during the period of double-digit inflation (when pay was capped) now get annuities that are higher than their former salaries. Retirees blame inflation for this, not the system.

* The annual payout to retirees is over $15 billion a year, plus $2.4 billion to survivors. Washigton area retirees get over a billion dollars a year. Retirees range from relatively well-off ex-supergraders to former clerks who live from month to month.

* The U.S. retirees whom politicians and the media see tend to be from the financial elite, the ones who have good incomes, second careers and summer homes. They rarely hobnob with ex-civil servants living in Georgia trailer parks, or apartments in College Park.

* Last election two candidates--Jimmy Carter and John Anderson--proposed limiting retirees to one COLA raise a year. Candidate Ronald Reagan pledged, in writing to the National Association of Retired Federal Employees, that he would not tamper with their twice-yearly adjustments if elected.

A followup poll by NARFE showed that better than 80 percent of its half million members voted for Reagan. The poll results were hardly public before Reagan proposed--and Congress agreed--to eliminate the September COLA adjustment, and give retirees one raise a year

Most feds realize their retirement plan is one of the nation's best. They help pay for it and, unlike Social Security, pay taxes on it. It is one reason they came into government. But a lot of them wish Congress and the White House would tackle the problem--inflation--not their catch-up-with-inflation raises.