Garfinckel, Brooks Brothers, Miller & Rhoads Inc. plans to expand its mail order business, particularly in its Brooks Brothers, Garfinckel's and Catherine's Stout Shoppes stores, company officials said yesterday.

"We plan to go after the mail order business in a serious way," company President Manuel B. Rosenberg said of the Brooks Brothers division. Rosenberg added that although Brooks Brothers has had a catalogue business for a long time, the company plans within the next three to four years to establish a separate mail order division for the store.

The mail order segment of the company's business is "relatively small," but is "growing dramatically," said David R. Waters, the company's chairman. bLast year the company hired a new vice president with expertise in the direct mail marketing business.

The Garfinckel division also will develope a large mail order business over the next five years, Rosenberg told a group of investors during the second day of a two-day conference sponsored by Legg Mason Wood Walker Inc. Mail order will become a major part of the store's volume and profits during the 1980s, he said.

The Catherine's division, which specializes in half sizes and large sizes of women's clothing, also will break into the mail order line and discount clothing market, Rosenberg said.

"We think that Catherine's will amplify its business significantly both through the discount business, which is happening right now, and through the mail order business which we will integrate at an appropriate time," he added.

Waters said that the company is on target with its goal of $600 million in sales by next year and $800 million by 1982.But he added that sales during the first quarter of this year have been "soft." He blamed sluggish sales in March on late spring weather and snow in some areas where the company has stores.

The company had a "weak midyear performance" last year, "but a strong fourth quarter" which ended in February, Waters said.

The company also plans to extend stores in its seven divisions, Waters said. For example, five new Miller & Rhoads stores, with outlets in Virginia and North Carolina, will be added by 1983. Four of those stores will be between 100,000 and 140,000 square feet in size, Rosenberg said.

Three new Garfinckel stores are planned, one in Georgetown and another in Annapolis. Rosenberg didn't say where the third store would be located.

The Brooks Brothers division, which recently expanded into the Tokyo market, will have two new stores in Japan soon as well as in New Orleans and Kansas City, Rosenberg said.