The battle over the number of cost-of-living raises federal and military retirees should get -- a $1 a day issue for 100,000 U.S. retirees here -- will go right down to the wire in Congress.

Both the Senate and House plan to recess this weekend for the GOP National convention and the Independence Day holiday. Congress is anxious to get a trimmed-down fiscal 1981 budget approved at least tenatively before it leaves.

The major item for government workers, postal employes, military personnel and retirees is what will happen to the COL raises they now get each March and September. The White House and budget committees of the Senate and House are pushing for a single annual adjustment that would occur in March.

Wednesday afternoon the Senate Governmental Affairs Committee approved a compromise COL cutback plan. It would drop the next cost-of-living raise, due this September, to comply with the Budget Committee resolution to save $500 million this year. The compromise would put the two COL payment system back into effect in 1981. Thus, retirees -- after skipping this September's raise -- would get a full catchup next March, and then another raise in September 1981 and two raises a year thereafter.

The Senate Budget Committee -- with White House backing -- voted to make the one-shot COL plan a permanent fixture. That is, that federal and military retirees be limited permanently to a single adjustment each year in March.

The question is whether backers of the double COL plan and the Governmental Affairs Committee compromise can win out, or whether the one-a-year COL raise will be made permanent.

Losing one of their two COL raises a year would cost the typical federal retiree about $1 a day.

Although the vote in favor of the compromise was 9 to 6 in the Governmental Affairs Committee, both its chairman, Abraham Ribicoff (D-Conn.), and ranking majority member, Tom Eagleton (D-Mo.), favor reducing federal and military retirees to one COL a year permanently.

Major committee backers of the compromise that is a restoration of the twice-yearly adjustments beginning in 1981 are Sens. Ted Stevens (R-Alaska), Charles Mathias (R-Md.), and David Pryor (D-Ark.).

Sen. James Sasser (D-Tenn.) voted against the compromise but only because he believes federal and military retirees should get two raises this year. He felt the compromise droppint the September 1980 raise) would weaken their case.

Meantime, retiree groups, federal and postal unions and military organizations are lining up what they hope will be enough votes to save the two COLs. Odds are against them.