James Walker spent 32 years as a government economist, and when he retired in 1967, he was a GS 14 making $17,000 a year. His government pension today -- the result of numerous cost-of-living adjustments -- is $2,400 a month, or about $29,000 a year.

Walker thinks that's too much, and he has been urging federal officials to temporarily halt further raises for himself and thousands of other retired workers.

"Since you seem to be serious in your attempt to reduce federal expenditures, I would like to help as much as possible," Walker has written David Stockman, the Reagan administration's budget chief. " . . . You must believe me when I say our retirement checks have been increased by much more than the actual cost of living increases justify."

Walker's plea, which comes on top of statements by some other well-placed federal retirees who say their annuities are "overly generous," reflects growing concern over civil service retirement benefits. The government's share of federal workers' pension costs has risen from $3.2 billion in fiscal 1972 to $18.2 billion in fiscal 1981.

His protest has exasperated spokesmen for federal worker groups who consider such criticism misguided and fear that cases like his only heighten ill-informed prejudices against federal workers.

"He's just not typical," says Steve Skardon, legislative secretary for the National Association of Retired Federal Employees (NARFE). "He's probably in the upper two percent of annuitants."

For his part, Walker, 73, doesn't think of himself has a crusader or point man in an assault on the federal retirement system. The former Labor Department economist approaches the debate from the perspective of a liberal, a member of Common Cause and the National Organization for Women and an independent voter who is a whole lot more comfortable around Democrats than around Republicans.

He says he finds his pension particularly disturbing in these days of Reaganomics because " . . . we are cutting back on basic programs for the poor that are badly needed, and here I am getting 11 percent more than I deserve and a hell of a lot more than I need."

Walker's pension puts him in the upper bracket of federal retirees. Of 1.3 million civilian retirees, according to the Office of Personnel Management, 57,470 are receiving pensions of at least $2,200 a month, or $26,400 a year. (Walker was, in fact, at the higher reaches of federal pay when he retired, and would now be making $53,600 a year if he were at the top level for a GS 14.)

Walker's pension grew over the years as the government increased his retirement pay to keep pace with inflation. Between 1965 and 1976, those cost-of-living adjustments (COLAs) were coming nearly every three months under a formula linked to the Consumer Price Index . An extra one percent was tacked on to each increase to compensate for the "lag" between rising prices and the adjustment of the annuity checks.

"The cost of living started going up so drastically, and I was getting the one percent on top of the one percent, and it just kept compounding," Walker says now.

An estimated 1.4 million retirees received their pension adjustments under this formula, which was eliminated in 1976. Now, because of a recent change, a single cost-of-living increase is given in March.

NARFE's Skardon emphasizes that most retirees are not nearly so well off as Walker, and neither are their survivors. "The average annual annuity for retirees is $11,000, and the average annuity to survivors is not even $6,000 -- and these people are suffering," Skardon says.

Still, concern persists over the federal retirement system.

The U.S. Chamber of Commerce points out that federal pensioners and retired military workers received an average of $7,700 and $8,000, respectively, in 1979 compared to the combined $6,800 in Social Security and pension benefits received by private sector retirees.

And Sen. Ted Stevens (R-Alaska) has introduced legislation that would convert the pension system into a modified civil service retirement plan with Social Security and a voluntary investment program.

His proposed system would not guarantee any minimum benefit, although Stevens says the plan has the potential for greater benefits. Opponents of the plan, largely unions representing federal employes, argue it would curtail their early retirement options and define contributions but not payouts.

Walker, meanwhile, has broadened his campaign beyond Stockman, writing other federal officials, President Reagan among them, and arguing that cost-of-living adjustments for people such as himself should be stopped until, in his view, the cost of living "catches up" to retirement pay.

In addition to his pension, Walker receives a small Social Security check each month and a "comfortable" income from his stock market investments, which he says he can afford to make because of his government pension.

Walker says he has not tried to return any of his pension because he doesn't trust this administration to spend it responsibly. He thinks the policies of Reaganomics "just took from the poor and gave it to the rich . . . the tax law was written for people like me."

Instead, Walker says, he makes charitable contributions to programs he cares about, such as day care, and keeps writing Republican and Democratic officials. Stockman, he says, never answered his letter. The White House turned a letter from him over to another agency, which promised--"months ago," Walker says -- to look into the matter.