Richmond Circuit Court Judge James E. Sheffield, emphatically denying he ever used his legal clients' money improperly, said today he will not withdraw as a nominee to become Virginia's first black federal judge.

Sheffield branded the allegations, contained in Internal Revenue Service records, as "totally inaccurate" and said they were disclosed as part of a smear campaign "geared toward assassinating my character and getting me to withdraw my name."

In his most detailed response to the charges, Sheffield added in an interview that he believes he has no choice but to continue to seek the judgeship to which President Carter nominated him last April.

"If I slink on off now, I'll have a cloud hanging over my name for the rest of my life," said Sheffield, whose nomination has evolved into a test of wills between the Carter administration, blacks and liberal Virginia Democrats on the one side, and Virginia Sen. Harry F. Byrd Jr. and white conservatives on the other.

Sheffield said he would not speculate on who was leaking the damaging allegations from the IRS records, copies of which are in possession of the Senate Judiciary Committee. But people close to the judge blamed Byrd's political allies among Republicans on the committee and its staff.

Byrd tonight denied any knowledge of the leaks and said he would continue to oppose Sheffield's nomination. "I've opposed him all I can and I will continue to oppose him all I can on the grounds that he was not the best qualified," said Byrd, who went on to say that the allegations against Sheffield "certainly add an extra demension."

The judge had abruptly halted his confirmation hearing before the committee two weeks ago just as he was due to be questioned about three IRS investigations of his tax returns over the last 10 years. Saying he had previously been denied access to IRS records of the investigations, Sheffield asked for and received a delay of the hearing until he had reviewed the files.

Highly placed sources said yesterday that the IRS files allege Sheffield took for personal use, or "commingled" with his own funds money from two clients, Church Hill Economic Development Corp. and Greater Richmond Development Corp. A source said Sheffield volunteered the information to the IRS as a defense against charges that he had failed to report a total of $108,000 as income in three tax returns between 1967 and 1975.

The source added that Sheffield told the IRS he repaid the money to his clients after deducting legal fees. The two now-defunct corporations, offshoots of a federally funded inner city development program, loaned money to minority businesses and kept between $500,000 and $1 million in escrow accounts over a period of a half dozen years, according to Willie E. Poe, former executive director of Church Hill Economic Development. Those accounts were overseen by the corporations' lawyers, including Sheffield.

Sheffield, who said he examined yesterday the report of the IRS agent in charge of investigating his case, denied he told the IRS he had commingled funds. he had said the report was ambiguously worded, and said he hoped the committee would call the agent to testify.

Sheffield confirmed his dispute with the IRS was over payment of back taxes, but said he believed the figure in dispute was less than $108,000. He said the IRS also challenged his bookkeeping for the escrow accounts but never accused him of criminal wrongdoing.

"They had a notion you could not keep two or three clients' cases in the same file or transfer money between escrow accounts," said Sheffield, who said he eventually reached a tax settlement with the IRS. He said he could not recall the amount he paid in back taxes.

"It was settled just like any other case -- they won on some of their claims and I won on others," said Sheffield. He noted that he later discussed the tax examinations in detail with Justice Department lawyers when they reviewed his background before he was nominated.

"I explained everything they asked me about and they were satisfied," said Sheffield. A spokesman said the department would refuse to comment, citing a policy against discussing pending judicial nominations.

Sheffield said the tax dispute was settled more than two years ago. He said he would this week ask the Judiciary Committee to reschedule his hearing for as soon as possible. A committee staff member said it would be at least two weeks before a new hearing could be held, a delay which could further jeopardize Sheffield's chances of being confirmed before the November elections.

Should Sheffield's patron, President Carter, lose the election, Senate Republicans can be expected to delay any remaining judicial nominations until Ronald Reagan takes office, a move that would doom Sheffield's confirmation. r

Byrd, a political Independent who retains the senatorial privileges of a Democrat, has opposed Sheffield's nomination because, he said, the judge was not one of 10 potential nominees suggested by two citizens' panels he appointed at Carter's request. Some of Byrd's political enemies claim racism is actually behind the senator's opposition and have noted that all 10 of Byrd's nominees were white males.