Richmond Circuit Judge James E. Sheffield yesterday blamed a mistake by an Internal Revenue Service agent and "an unfortunate communications problem" for tax troubles that have stalled his confirmation for a federal judgeship in Virginia.

Legal documents filed yesterday with the Senate Judiciary Committee, six weeks after Sheffield abruptly halted his own Aug. 26 confirmation hearing, included an eight-page sworn affidavit by the judge in which he denied using funds of legal clients as his own and denied underreporting his income on federal tax returns.

Congressional sources said yesterday the delay means it is almost certain that Sheffield's nomination will not be considered until after the Nov. 4 election. If President Carter loses the election, these sources said, it is likely that committee Republicans would request delays that would prevent Sheffield's nomination from being approved.

"The ultimate question is not whether the (IRS) agent correctly or incorrectly understood what Judge Sheffield told him," said Burt Wides, an investigator for the Senate Judiciary Committee. "The ultimate question is what was Sheffield's conduct and what do the senators think?"

IRS officials yesterday declined to comment on Sheffield's statements, which were made public at press conferences in Washington and Richmond by two lawyers who are advising the 43-year-old judge, the only black to sit on a Virginia court of record.

"The judge was talking on AM and the agents were listening on FM," said Richmond attorney James R. Wrenn Jr., former chairman of the disciplinary committee of the Virginia State Bar Association, who added that Sheffield "is not trying to point a finger at anybody."

Samuel C. Thompson Jr., a University of Virginia law school professor and expert in federal tax law, said that during a 1975 audit of the judge's 1973 tax return, the IRS agent, who was not named, "erroneously inferred" that Sheffield had improperly used clients' funds, or in legal terms, commingled them, before performing legal services. Those statements were part of Sheffield's IRS file.

Thompson said that at the time, Sheffield acted as both attorney and disbursing agent for two now-defunct corporations -- Church Hill Economic Development Corp. and Greater Richmond Development Corp. -- whose purpose was to provide housing for blacks in the inner city.

As disbursing agent, Thompson said, the judge wrote himself checks for legal fees on the escrow accounts of the corporations, fees to which Thompson said Sheffield was legally entitled although the judge had no written fee agreement with the corporations.

Thompson said that during a Sept. 25 interview with the IRS agent, the agent acknowledged to Thompson that Sheffield had never used the word "mingling," nor had he said he had withdrawn fees before performing legal services.

In his affidavit, Sheffield denied a Sept. 9 report in The Washington Post, quoting a highly placed source as saying that Sheffield had voluntarily told the IRS he once commingled the corporations' funds as a defense against charges he had failed to report a total of $108,000 as income between 1967 and 1975.

Thompson also denied that Sheffield had failed to report income and said the allegation was the result of another misunderstanding that arose when Sheffield voluntarily reduced his legal fees to the corporations and credited the remainder of the original fee to other accounts.

The fees in question, according to Sheffield's affidavit, were $6,400 and $891.78.

"He did something one would expect and hope lawyers would do -- make adjustments in their fees," said Wrenn, who called the controversy surrounding Sheffield's finances "a terrible smear."

Neither of the two corporations ever complained about Sheffield's handling of the accounts, according to the affidavit.

Sheffield's nomination has been dogged by political problems from the start, most notably the opposition of Harry F. Byrd Jr.(Ind.), Virginia's senior senator and heir to a political dynasty that civil rights leaders characterize as historically racist. Byrd has insisted his opposition has nothing to do with Sheffield's race or with Jimmy Carter's goal of appointing blacks and women to Virginia's traditionally all-white, all-male judiciary.