Boxing promoter Don Elbaum, facing charges that he failed to pay federal income tax from 1983 through 1985, said yesterday he did not file returns for those years, but added that he did not earn the $570,000 the Internal Revenue Service claims he did over that time.

"I want to see where they came up with the figures," Elbaum said. "There's no way I made that kind of money. I wish I made that kind of money."

Assistant U.S. Attorney Diane Weeks, who is handling the case, said Elbaum earned gross income of $338,671 in 1983; $148,943 in 1984; and $82,188 in 1985. She declined to comment as to whether Elbaum had filed tax returns in the years since, pending results of a continuing investigation by the IRS.

Elbaum, 50, who until earlier this year promoted the fights of International Boxing Federation welterweight champion Simon Brown, has been charged with three counts of willful failure to file tax returns by the U.S. District Court in New Jersey.

The promoter faces a maximum of three years in prison and fines of $225,000 if convicted. A trial date has been set for Dec. 11 in Trenton before Judge Garrett Brown. Charges were filed Sept. 7, according to a story Tuesday in the Journal newspapers, when Elbaum was still a resident of Longport, N.J. He lists as his legal address the Holiday Inn Crowne Plaza in Rockville, where he promotes monthly boxing cards, including one last night.

"This is something {the investigation} that's been going on for a while now," Elbaum said. "Right now I just want to leave it in the attorney's hands."

Elbaum's attorney, Myron Beldock, could not be reached to comment, but he told the Journal Tuesday that Elbaum would plead guilty to a misdemeanor for one of the three counts with the hope the other two would be dropped.

Elbaum's former partner, Teddy Menas, pleaded guilty Oct. 24 to similar charges. The two promoted fight cards at the Tropicana Hotel and Casino in Atlantic City from 1981 to 1986. Menas is charged with one count dealing with his 1983 tax return, and prosecutors dropped charges of failing to file returns in 1984 and 1985.

Crowne Plaza General Manager Steve Ball said fight cards will continue at the Rockville hotel, with or without Elbaum as promoter.

Elbaum would not comment on a Journal story saying Brown, who blossomed in Washington and lives in Germantown, has joined the camp of promoter Don King and will fight an opponent to be named on the undercard of the Mike Tyson-Alex Stewart bout Dec. 8 in Atlantic City. A Supreme Court judge in New York City last week gave Brown permission to pursue fights with other promoters.

Elbaum and partner Alan Baboian have a $2.5 million countersuit pending against Brown, claiming breach of contract. New York Supreme Court Judge Beverly Cohen is expected to reach a verdict this week or next, Elbaum said.