Garfinckel, Brooks Brothers, Miller & Rhoades Inc. has tried unsuccessfully to buy out its biggest stockholder and avoid a public takeover fight.

But Wickes Companies Inc. rejected Garfinckel's offer, setting off a legal shootout that could lead to Wickes or someone else gaining control of Garfinckel's diversified retailing empire.

Details of the Garfinckel-Wickes fight are emerging from a pair of lawsuits that the two companies slapped on each other this week. Garfinckel sued Wickes in U.S. District Court here Wednesday, without knowing that Wickes had beaten them to the draw, by filing its own suit in New York late Tuesday.

The two legal actions probably will be combined into one definitive lawsuit, spokesmen for the companies said yesterday. Both sides reported that talks between the two companies have broken down since the legal actions were taken and no new negotiations are scheduled.

Wickes owns almost 21 percent of Garfinckel's stock. Wickes got the shares when it purchased Gamble-Skogmo Inc. of Minneapolis last year; Gambles had bought the shares in 1978 in an unsuccessful effort to buy control of Garfinckel.

Garfinckel blocked that takeover with a series of legal maneuivers that ended when Gambles signed an agreement not to buy any more Garfinckel stock or sell its shares without offering them to Garfinckel first.

In its lawsuit, Wickes claims it is not bound by the settlement between Garfinckel and Gambles, because it was not involved in the agreement.

Wickes attorneys asked the court to declare the agreement invalid and to prohibit Garfinckel "from interfering with Wickes' right to purchase Garfinckel common stock."

Garfinckel's lawsuit in Washington asks the courts here to uphold the agreement and stop Wickes from buying or selling stock.