Modest gains in profitability were reported yesterday by major specialty and department store retail companies for 1980 and the final quarter that included Christmas shopping days. Profit margins generally were reduced because of abnormally heavy sales promotions to move merchandise off the shelves and racks, according to retail executives.

Garfinckel, Brooks Brothers, Miller & Rhodes, for example, listed nationwide sales at its diverse stores of $160.5 million in the quarter ended Jan. 31, an increase of 15 percent from the same period a year earlier. But profits were up just 7 percent to $9 million ($2.06 a share) from $8.4 million ($1.90).

Higher sales and earnings also were reported by officials of Associated Dry Goods and by the Federated and May department store companies, the two largest in the nation.

David Waters, chairman of Washington-based Garfinckel, Brooks Brothers, said sales were particularly strong in November and December, followed by a softening retail climate in January.

For the full fiscal year ended Jan. 31, Garfinckel, Brooks Brothers sales jumped 12 percent to $472 million as profits edged slightly higher to $14.8 million ($3.33) from operating profits of $14.6 million ($3.24). In the year-earlier period, a one-time gain from selling the Joseph R. Harris division had produced net income of $15.3 million ($3.40).

Waters said specialty stores such as the Garfinckel division in Washington and the Brooks Brothers menswear operations showed stronger gains than department stores in the recent year.

Federated Department Stores Inc., owner of Bloomingdale's and I. Magnin, listed profits from operations in the year ended Jan. 31 of $219.6 million ($4.54 a share) compared with $203.2 million ($4.21) a year ago, an increase of 8 percent as sales rose 8 1/2 percent to $6.3 billion.

Fourth-quarter net income was $122 million ($2.52) compared with $104 million ($2.16) a year ago as sales increased 9 percent to $2.1 billion. Officers of the largest department store firm in the country said sales were somewhat stronger than expected at their big stores in the final quarter of the year while mass merchandising was affected by the sales promotional atmosphere of many markets.

May Department Stores Co., parent firm of Hecht's in the Washington and Baltimore markets, earned $117 million ($4.01 a share) in the year ended Jan. 31 compared with $113 million ($3.90) a year earlier while sales rose 6 1/2 percent to $3.15 billion. In the final quarter, May profits were $67 million ($2.30) vs. $60 million ($2.08) as sales topped $1 billion for the first time in a three-month period at 1.06 billion compared with $991 million a year ago.

Associated Dry Goods, which owns Lord & Taylor specialty stores and Stewart's of Baltimore, listed fourth-quarter profits of $36 million ($2.68 a share) compared with $32.7 million ($2.42) a year ago as sales rose to $673 million from $600 million. In the year ended Jan. 31, Associated earnings totaled $5.52 million ($3.86) vs. $43.9 million ($3.25) as sales increased to $1.95 billion from $1.78 billion.