Profits of Garfinckel, Brooks Brothers, Miller & Rhoads jumped at least 20 percent in the first quarter and the growing national retail company is "on target" to achieve growth plans that see annual sales of $600 million by 1981 and $800 million two years after that.

These were among the announcements of Garfinckel Chairman David Waters at the firm's annual meeting here yesterday-a two-hour session marked by union complaints about bargaining at its New Jersey shirt plant and continued bitterness about the move last year by another retail firm to buy a large share of the Washington company's stock.

But Waters hinted that the unfriendly and costly takeover fight with Gamble-Skogmo Inc., of Minneapolis, might soon be resolved.

In response to repeated questioning by professional stockholded Evelyn Y. Davis, the Garfinckel chairman revealed that the local company is "having out-of-court discussions with Gamble-Skogmo . . . we're talking . . . but I want the lawyers (there) to protect the stockholders . . ."

Davis had asked about the legal costs involved with the Gamble-Skogmo fight and complained that "lawyers like nothing better" but to drag out such ligitation to fatten their own fees. Waters told the stockholders that in 1978, the Gamble-Skogmo fight cost the Garfinckel firm $179,500 in legal fees out of a total bill for outside legal services of $374,900.

Gamble-Skogmo, which operates general merchandise, variety and specialty stores, bought 20 percent of Garfinckel stock last year from J. Robert Harris Jr. and other members of the Harris family-touching off a series of lawsuits including a Garfinckel antitrust case in Delaware.

That suit alleges substantially lessened retail competition in violation of the Clayton Act if the Minneapolis firm is allowed to retain and expand its ownership of the Washington company.

In his report to the meeting and in response to stockholders' questions, Waters revealed a substantial number of new developments and offered limited comments on complaints by the Amalgamated Clothing and Textile Workers Union (AFL-CIO) about conditions at a Brooks Brothers shirt factory in Paterson, N.J.

Among the subjects covered, Waters said:

Although final first-quarter results were not available, profits from continuing operations will be up 20 percent from an "exceptional" period last year, when earnings were $1.4 million (30 cents a share). Sales rose about 12 percent from $77 million last year despite snowstorms in February, which cut into volume.

Garfinckel management takes the position that Brooks Brothers did not commit "unfair practices" at the Paterson factory and "we would welcome a union election if one should come." In a formal complaint, the Newark regional director of National Labor Relations Board has accused Brooks Brothers of firing union sympathizers and threatening workers; hearings on the charges have been scheduled next month.

The Brooks Brothers division and the Washington-based Garfinckel specialty stores are moving to expand substantially their mail-order businesses, which should benefit from new concerns about gasoline shortages and increased consumer shopping by catalogue.

A substantial increase in capitol spending is planned this year, which may require some form of outside financing in 1979 or early 1980. The firm will spend $20 million this year ($8.8 million for new stores and the balance for refurbishing and fixtures), compared with $8 million in 1978 and $3.7 million in 1977.

The Brocks Brothers division is planning a major new store in downtown Washington-"pretty close" to the intersection of 19th and K Streets NW. An existing store at Connecticut and L Streets is located on a site where a new retail complex is planned but Garfnckel has a lease there with seven years remaining and the building "is not coming down until we move."

A recent agreement with a Japanese firm will lead to the opening of eight Brooks Brothers stores in the Tokyo area by 1985, with an initial store opened later this year. But Canadian and British tariffs and exchange rates have ruled out stores in those countries as "not economically right."

No exact price has yet been established for the sale of the Joseph R. Harris stores to Petrie Stores Corp., but it will be less than $10 million and have "no major impact on earnings." The Harris stores have been operating at a loss (31 cents a share last year) and management decided it would require too much effort to turn the retail units around given prospects for only gradual profit growth.

The Garfinckel division, with new stores planned in Georgetown, Fairfax County and Annapolis, is drawing up plans to expand outside its current Washington area market. Miller & Rhoads, based in Richmond, will add three department stores in 1980 in Virginia Beach, Lynchburg and Charlottesville-adding 20 percent to that division's retail space in one year and laying the groundwork for additional expansion thereafter.

Of shares voted at the meeting, 3.9 percent were in favor of a Davis motion calling for disclosure of law firm payments.

And, in a related development, Gamble-Skogmo announced yesterday that it will neither make a profit nor suffer a loss on the proposed sale of its Canadian stores to Cavendish Investing Group for $80 million. Gamble wants to unload 128 stores and 6 mass merchandising outlets because of Canadian taxes, exchange rate problems and a difficulty of expanding in Canada, the firm stated. CAPTION: Picture 1, DAVID WATERS . . . mail-order expansions; Picture 2, Members of clothing workers union protest bargaining at New Jersey shirt plant by picketing site of annual Garfinkel's meeting that was held yesterday. By James M. Thresher-The Washington Post; Picture 3, Corporate gadfly Evelyn Y. Davis shakes hands with Garfinkel's Chairman David Waters following yesterday's two-hour stockholders meeting held here. By James M. Thresher-The Washington Post