If the government puts a ceiling on the annual Social Security cost-of-living increase, the average retiree now on the rolls would lose as much as $9,800 in benefits by 1990 and receive a monthly check that buys 25 percent less than today's check, according to a study released yesterday by the House Committee on Aging.

Chairman Claude Pepper (D-Fla.) said any of three major proposals under consideration by White House and congressional budget negotiators to "cap" the annual cost-of-living adjustment (COLA) would be "devastating" to the aged and push at least 1.2 million elderly into poverty by 1990.

The study shows that the average retiree receives $385 a month and that under existing law would receive $657 monthly by 1990 with COLA increases.

However, one proposal under discussion would hold the increase to $507, the second to $509 and the third to $554, according to committee calculations. In each case, the benefit would be worth less in purchasing power than the $385 is today.

The dispute about whether to cap the COLA, under which monthly benefits are increased each July to keep pace with the consumer price index, is one of the most bitter issues in the budget negotiations.

House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) and President Reagan have said they oppose capping the COLA.

The committee study calculated how each of the three plans would alter the benefits of the average worker on the rolls last October. It showed that:

* The proposal by Senate Budget Committee Chairman Pete V. Domenici (R-N.M.) and Sen. Ernest F. Hollings (D-S.C.) would be harshest, canceling the COLA this year and limiting it to 3 percentage points less than the CPI increase each year but no less than 3 percent.

* An option offered by Budget Director David A. Stockman would limit any COLA increase to 4 percent, postpone the scheduled July 3 increase for three months and pay COLA increases every 15 months instead of annually.

* The Congressional Budget Office plan would have the least impact, holding the annual COLA to two-thirds of the actual CPI increase each year.