An apparent tentative agreement that would have brought holdout Lydell Mitchell back to the Baltimore Colt camp was rejected late Monday by club owner Robert Irsay because of his concern over racism charges leveled against the club by the running back, it was learned yesterday.

A meeting Monday reportedly had left the two parties $50,000 apart on a three-year contract in which Mitchell is seeking $800,000 sources said.

But Irsay rejected the near-agreement that had been tentatively reached by his cogeneral managers.

Charles Sullivan, chairman of the NFL, Management Council, who attended Monday's meeting, said yesterday that Irsay's main concern was the racial charge.

"Irsay is very upset because he has always conducted his business (Pro football and an air-conditioning business in Skokie, Ill.) as a positive force in race relations," Sullivan said.

"He was the primary fund raiser for the Better Boys Foundation of Chicago, for the benefit of underprivileged youth, mostly blacks," Sullivan added.

Mitchell, in the option year of a contract that pays an estimated $99,000 a year, has refused to report to camp while trying to negotiate a new pact in the $300,000 a year range. He has filed a grievance with the NFL Players Association, contending the Colts were practicing racial discrimination. He also contended the club was penalizing him improperty by fining him $500 a day while he remains out of camp.

Previously, Irsay sought the help of civil rights leader the Rev. Jesse Jackson to resolve the impasse with Mitchel. Jackson did not attend the meeting in Washington.

One report was that a Colt representative in a prior negotiating session with Mitchell and his attorney said, "Mitchell would be the second highest paid black on the Colts' squad."

Another report was that the Colt representative insisted he said "Mitchell would be the second highest paid back," behind quarterback Bert Jones.

A source said the Colts in Monday's session flatly denied using any racial language previously, but Mitchell's attorney, Lee Goldberg of Pittsburg, repeated the charge.

Both Sullivan and Goldberg said yesterday that those who attended Monday's meeting probably would reconvene today or tomorrow.

Ed Garvey, executive director of the NFL Players Association, said of the meeting on Monday among two Colt officials, Mitchell his attorney and the National Football League player Club Relations Committee. "We had a compromise that would have made it possible for Lydell to go back to the team and defuse the situation entirely.

"Then at midnight, Irsay informed us his representatives (Dick Srymanski and Ernie Accorsi) had no authority to speak for him and he rejected the agreement. It's almost like a fraud."

Mitchell said, "I'm just shocked. I thought we had a settlement last night, and then I find out that the owners won't live up to it."

Later he said, "Maybe it would be best for all concerned if I were to be traded. I think, though, that anything can be resolved."

Irsay did not respond to telephone calls to his office yesterday, but NFL Management Council chairman Sullivan said. "What basically happened him you're crazy for throwing that was that a tentative agreement was reached, subject to Mr. Irsay's approval, and he didn't give his approval.

"Srymanski (executive vice president and general manager) and Accocsi (assistant general manager) recommended the tentative agreement. The player (Mitchell) and Garvey were advised the agreement required the approval by Isray. The player was not that interested in it either."

Asked yesterday if the parties were near a salary agreement. Mitchell's attorney said, "I wouldn't say that. I can't comment. It is a question of agreeing on the grievances AND the contractual relationships."

Resisting giving details about the racial charge, he said, "A serious charge has been made . . . and not frivolously. Does it come under the collective bargaining agreement (for resolution) or a court of law? I made the allegations, as Lydell's lawyer. The committee has the power to resolve it."

Was it the sort of racial situation that could be solved by an apology?