A federal judge refused yesterday to let the defense attorney for D.C. mayoral aide Joseph P. Yeldell withdraw from representing the city official at his bribery trial in September.

U.S. District Judge Gerhard A. Gesell denied the motion by Curtis R. Smoothers, Yeldell's lawyer, pointing out the "imminence of trial" and the lack of any substitute attorney able to meet the trial schedule on Yeldell's behalf in the complicated fraud case.

Yeldell and multimillionaire developed Deminic F. Antonelli Jr. are scheduled to go to trial Sept. 18 on federal conspiracy and bribery charges in connection with yeldell's awarding of a city lease to a building owned by Antonelli. In return, yeldell is alleged to have received a $33,000 loan at favorable rates and other financial favors from Antonelli.

The surprise motion by Smothers last Friday to withdraw from representing Yeldell came amid reports in the Washington legal community that Yeldell had not been paying his attorneys for some time and had been unwilling to discuss a payment schedule with them.

Smothers and a former assiciate, Gary Myers, had represented Yeldell for about 11/2 years in his various troubles before city and federal agencies. Smothers gave no reason for wanting to withdraw from the case, stating only that he had told Yeldell to find a new attorney.

Smothers declined to comment late yesterday.

In addition to the reports that Yeldell had made only minuscule payments, if any, to Smothers for legal work, there were reports from some sources saying they understood that Yeldell's attorneys had expressed a concern about what they said was his lack of candor with them on certain issues.

Instead of holding a hearing to determine why Smothers wanted to withdraw, Gesell issued a brief ruling strictly reading the court's rules as precluding Smothers from getting out of the case unless a new attorney who can be ready for trial Sept. 18 comes in for Yeldell.

Several lawyers have said Yeldell may have difficulty finding a competent attorney to begin preparing for a September trial after Yeldell's troubles with Smothers have surfaced, even if Smothers is eventually allowered to withdraw.

The trial could take more than a month; the government has said it has more than 50 witnesses. As part of the pretrial discovery phase, prosecutors already had turned over to Smothers 30.000 pages of documents that had been accumulated during the 16-month federal investigation.

Somthers and Myers have represented Yeldell since November 1976, when Yeldell's actions as director of the D.C. Department of Human Resources were called into question. News accounts at the time listed allegations of nepotism, croynism and leasing abuses by his agency.

Smothers and Myers became Yeldell's lawyers at the height of criticisms against him. By April 1977 they had guided him through several city government investigations. Those investigations failed to establish any wrongdoing by Yeldell, and Mayor Walter E. Washington subsequently named him to be the No. 3 man in his administration.

A federal investigation of Yeldell's leasing activites continued, however, and Yeldell and Antonelli were indicted last month.

After the indictment, Mayor Washington and Yeldell agreed that Yeldell should be placed on leave with pay - paid vaction - while he prepared his defense. Yeldell and Antonelli pleaded innocent to the charges.