Garfinkel, Brooks Brothers, Miller & Rhoads, the largest general merchandise retail firm based here, yesterday announced plans to acquire Ann Taylor, Inc., a specialty retailer of women's wear, for about $5 million.

The Taylor chain, based in New Haven and New York, has been regarded as a trend-setter for high fashion contemporary clothing since it was founded in 1954. It operates 27 stores - including branches here at Tysons Corner Center, Georgetown and White Flint mall.

Garfinckel, Brooks Brothers already operates more than 90 department and specialty clothing stores in 16 states and the District with annual revenues of some $300 million.

According to David R. Waters, president of the Washington company, Garfinckel's will pay $4.2 million in cash and $800,000 of Garfinckel's stock to shareholders of Ann Taylor in the proposed takeover, "subject to certain adjustments."

Richard L. Liebeskind, founder and owner of 95 per cent of Ann Taylor stock, will continue as president of the business, which will become a subsidiary of Garfinckel's. The acquisition is subject to execution of a definitive agreement.

In an interview yesterday, Liebeskind said talks with the Garfinckel firm began last February and March and were conducted through a broker. He said that with the additional financial resources of the Washington firm, Ann Taylor will have the capital needed for rapid expansion.

Over the next two years, he said, Ann Taylor will add from four to six new stores a year. After that, expansion will be accelerated. Liebeskind said he is seeking sites on lower Connecticut Avenue NW and in the Friendship Heights area for new branches in the Washington area.

In the fiscal year ended last July 31, Ann Taylor had sales of $14 million; for the first nine months of the current fiscal period, sales were nearly $11 million.

Waters said he will work of closely with Liebeskind to "aggressively" add Ann Taylor locations "into every sophisticated shopping area of the country." Currently, the Taylor shops are located in Connecticut, Boston, New York and Chicago, as well as Washington.

Although the past year has been a difficult one for retailers, Ann Taylor's performance has been improving in the last couple of quarters, Liebeskind said.

Ann Taylor has been profitable since 1956, earning 9 to 10 per cent of sales before taxes - a rate of profitability that is unusually high for a retail business (Garfinckel's pre-tax profits were slightly more than 6 per cent in the last fiscal year).

The Garfinckel business was started here in 1905 by Julius Garfinckel but its oldest subsidiary, Brooks Brothers, dates back to 1818. Other divisions of the Washington firm include the 27-store Joseph R. Harris women's specialty stores in the mid-Atlantic states: Garfinckel's in Washington; Miller & Rhoads of Richmond; Miller's of Knoxville; Miller Bros. Co., with stores in Georgia, Tennessee and Virginia; and Harzfeld's with stores in Missouri, Kansas and Oklahoma.