Claiming promoter Don Elbaum and manager Allen Baboian still owe him $28,000 from his last bout, International Boxing Federation welterweight titleholder Simon Brown is refusing to sign a contract for a three-fight package that would eventually match him against four-time world champion Roberto Duran, possibly at RFK Stadium.

"I am not fighting for those {Elbaum and Baboian} anymore," said Brown, who broke with his camp in August. "A package deal like some Golden Gloves tournament is not right for Simon. I want to fight Duran, but straight on."

Elbaum said yesterday he worked out the contract with Duran's promoter, Michael Acri, and sent it to Brown for his signature. Brown, reached at home, says he gave the contract immediately to adviser James Cooks, who called the deal a "ploy by Elbaum to assert control over this young man's life."

Brown also said Elbaum and Baboian still owe him following his most recent title defense, an April 1 knockout of Tyrone Trice.

Elbaum, however, denied the charge. "I've heard that talk before and I don't see how we owe him that money," he said.

The first bout in the package would be either Nov. 27 or Dec. 4 at the 10,000-seat D.C. Armory, Elbaum said, with Brown and Duran in separate bouts against opponents to be determined.

Brown (32-1, 24 knockouts) would defend his IBF title in the second bout, in late January or early February in the District against one of three opponents: former IBF junior welterweight champion and No. 1 contender James "Buddy" McGirt, No. 2 contender Glenwood Brown or ex-World Boxing Council junior welterweight champion Roger Mayweather. Duran would fight on the undercard.

In the final bout, in late May or mid-June, Brown and Duran would meet in a 15-round middleweight bout. Elbaum said he was looking at Madison Square Garden in New York or RFK Stadium as a possible site.

Duran, 39, has fought professionally since 1967. The Panamanian-born fighter (85-8, 60 knockouts) has held titles in four weight classes: lightweight, welterweight, junior middleweight and middleweight. He lost the World Boxing Association middleweight crown Dec. 7 to Sugar Ray Leonard in his most recent bout.

Elbaum said he and Brown had discussed Duran as an opponent to "put him over the top in his career. Simon eventually wants to fight Leonard, but the only way to get to him is through Duran," the promoter said.

Brown agreed, saying he does want to fight Duran, but not with Elbaum and Baboian's association.

Elbaum said late yesterday that he had not spoken to Brown about the matter and hopes it can be resolved. "There's no Duran fight without us. He can blow the fight he's always wanted."