A fight is brewing between the management of Columbia First Federal Savings & Loan Association in Washington and its biggest shareholder over the thrift's direction and profitability.

Harold N. Goldsmith of Baltimore, chairman of Merry-Go-Round Enterprises Inc., a retail clothing chain, has launched a campaign to unseat three directors of Columbia First in a dramatic move to extend his influence on the S&L's management.

Goldsmith -- who owns 240,600 shares, or 9.8 percent, of Columbia First stock -- has notified the Federal Home Loan Bank Board that he and two associates will seek election to the 11-member Columbia First board at the thrift's Jan. 26 annual meeting.

The management of Columbia First is expected to resist Goldsmith's efforts. Goldsmith reportedly will wage his campaign on the argument that the board needs more active directors and that the thrift's financial performance can be improved.

Dewitt T. Hartwell, president of Columbia First, declined to comment on Goldsmith's move.

Columbia First, with assets of $1.8 billion, has 14 branches in the District, five in Virginia and six in Maryland.

While a Goldsmith victory would not give him control of the board, it would represent a foot in the door and an opportunity to oppose present management. It also would make it easier for Goldsmith supporters to go after the four board seats up for election in 1989.

Joining Goldsmith in the proxy fight are David C. Daneker, a Baltimore attorney, and Barry H. Stern, president of Stern and Moran Properties, a real estate development company in Washington.

They will oppose current directors C. Gay Harrell Jr., executive vice president of Columbia Federal; James M. Jacobsen, owner of Jacobsen Brothers Inc. and Jacobsen Properties; and Flaxie M. Pinkett, president of John R. Pinkett Inc., a Washington real estate company.

Goldsmith, whose office said he was out of the country yesterday, also is president of Eastern Savings Bank in Timonium, Md.

Merry-Go-Round Enterprises of Towson is a $200-million-a-year company in which Goldsmith has a large stake.

Goldsmith was described by acquaintances as a wealthy 45-year-old businessman with a variety of interests.

Goldsmith's interest in Columbia Federal was first reported in June, when he told bank regulators that he had bought 126,000 shares at a cost of $1.9 million for the purpose of "capital appreciation." Goldsmith has steadily increased his stake in the company in the months since, and at yesterday's price of $16.75 bid, Goldsmith's 240,600 shares are worth $4 million.

Columbia First shares, which traded as high as $20 prior to the market collapse on Oct. 19, slipped to the $14 to $15 area after the plunge but began to recover last week. Columbia First went public two years ago, opening at $10 a share.

Since then, the thrift has attracted interest from several major investors, including Diana Corp. of Milwaukee, which owns 239,150 shares, or 9.7 percent, of the stock.