The already-heated battle over three board seats at Columbia First Federal Savings & Loan of Washington began to sizzle yesterday when Chairman Dewitt T. Hartwell accused challenger Harold N. Goldsmith of potential conflict of interest because Goldsmith holds investments in three other local financial institutions.

Goldsmith and two colleagues are trying to unseat three incumbents on the 11-member Columbia First Board at the thrift's annual meeting Tuesday. They argue that the thrift has been poorly run over the past few years, resulting in disappointing financial results.

Goldsmith, who owns 10 percent of Columbia First stock, is chairman of Merry-Go-Round Enterprises, a Towson-based clothing retailer, and is president of the Eastern Savings Bank of Timonium, Md.

Hartwell, in a letter to shareholders, said, "Goldsmith's substantial investments in a total of four thrift institutions in the same market area could in our view present him with a significant conflict of interest. We believe that these potential conflicts could seriously impair Goldsmith's ability to fulfill his fiduciary responsibilities as a director of the association."

Goldsmith denied the charge. "I have no conflict of interest," he said. "All I have is investments in other institutions."

Hartwell said Goldsmith controlled Eastern Savings Bank, held 10 percent of Columbia First Federal stock, 8 percent of the stock of Yorkridge-Calvert Savings and Loan Association and 5.2 percent of the stock of Maryland Federal Savings and Loan Association, all located in the Washington-Baltimore area.

The Goldsmith-Hartwell struggle appeared to take a turn in favor of Columbia First on Jan. 13, when Diana Corp. of Milwaukee, a major holder of the thrift's stock, made a deal with Columbia First executives to support the incumbent board members. In return, Diana won the thrift institution's support of Diana's desire to boost its stake beyond the 10 percent limit to 24.9 percent of Columbia First stock.

What was not immediately revealed by management was that the deal with Diana was a 14-month, two-election agreement, in which Diana would vote for management candidates in this year's and next year's election.

Hartwell's explanation was that a one-election agreement would give Diana, once it got its 24.9 percent, an opportunity to vote out the present management at the 1989 meeting. By getting Diana's support for two years, Hartwell said, "It will give us a chance to show what we can do."

The two-election nature of the deal was reported yesterday as part of the letter in which Hartwell asked shareholders to oppose Goldsmith's bid.

Goldsmith denounced the thrift's agreement with Diana. In an advertisement in The Washington Post, Goldsmith declared, "This agreement is, in our opinion, no more than an attempt to acquire votes and further insulate current management, all without the voice or approval of the stockholders."

Hartwell's letter quoted from a recent Goldsmith letter to stockholders. In that letter Goldsmith criticized the thrift's financial performance in the fourth quarter of 1987 as "disappointing" and said, "A new voice should be heard on the board of directors."

Hartwell responded that "although Goldsmith has been a substantial investor in the association's stock for more than a year, he has never once contacted management to express any concerns or criticisms about the association."

Running with Goldsmith are David C. Daneker, a Baltimore attorney, and Barry H. Stern, president of Stern and Moran Properties, a real estate development company in Washington.

They are opposing current directors C. Gay Harrell Jr., executive vice president of Columbia First; James M. Jacobsen, owner of Jacobsen Bros. Inc. and Jacobsen Properties, and Flaxie M. Pinkett, president of John R. Pinkett Inc., a Washington real estate company.