CIA Director William J. Casey's brain tumor could have affected his testimony to congressional committees about the Iran arms deals in the weeks before last month's brain surgery, medical experts said yesterday.

Sources have said that a draft report on the arms affair by the Senate Select Committee on Intelligence initially criticized Casey's testimony before the committee as "less than candid." But after his hospitalization last month, that criticism was amended on the theory that Casey, 73, might have been suffering from the tumor's effects.

Medical experts interviewed yesterday said it is likely that the Central Intelligence Agency chief's tumor existed for months before it was discovered and might have affected such brain functions as short-term memory, concentration, processing of new information or ability to answer complex questions in the weeks before he was hospitalized.

"I think a person who . . . is carrying that tumor, you just can't say that he was at his best and that judgment and recall were not affected by it," said Dr. Nicholas Zervas, chief of neurosurgery at Massachusetts General Hospital. "Even if he knew, he might not have been able to remember."

Casey testified before congressional committees on Nov. 21 and Dec. 10. He suffered a seizure and was rushed to Georgetown University Medical Center on Dec. 15, the day before he was scheduled to appear again before the Senate intelligence committee. He had a second seizure at the hospital.

He underwent brain surgery for the tumor, a lymphoma, on Dec. 18.

He remains in stable condition at Georgetown and is undergoing radiation therapy. A hospital statement last week said Casey is having difficulty speaking and has some weakness on the right side of his body, but that both conditions are slowly improving.

Experts interviewed said that, depending upon the size of the brain tumor and its exact location, Casey may have suffered no impairment of his mental capacities before the seizures. On the other hand, he may have had subtle impairment of functions such as memory, concentration or speech for months.

Someone with a tumor such as Casey's might had have trouble keeping track of dates or absorbing details, according to Dr. William R. Shapiro, professor of neurosurgery at the Memorial Sloan-Kettering Cancer Center in New York. He said that recent memory, involving events of weeks or months preceding the illness, would be more likely to be affected than memory of events many months or years ago.

He added that the type of tumor Casey had, a lymphoma, is particularly insidious because it grows like a network within brain tissue rather than in a ball or lump. "It can kill off brain" tissue, he said. "But that, curiously enough, can be compensated for. We're born with an awful lot of brain we don't use."

As a result, the tumor can induce gradual, subtle changes in personality or mental functioning that may go unnoticed by associates or may be attributed to stress until there is a dramatic symptom, such as a seizure, he said.

Zervas said that when a brain tumor is discovered, "you can usually find a six month-interval before, when the family begins to question certain things." He said changes in personality or functioning are more common when the patient is elderly.

He added that besides damaging nerve tissue, tumors sometimes provoke electrical disturbances within the brain, causing seizures too mild to be noticed but that interfere with thought processes.

The Associated Press reported yesterday that a handful of people, including FBI Director William H. Webster, have been approached as possible replacements for Casey but that the search for a successor was suspended after President Reagan objected that Casey's recovery might be impeded by hearing news accounts of it.

Casey has expressed a wish to return to the CIA at least for a "symbolic period," according to an associate.

White House officials said yesterday that Reagan is not seeking a successor to Casey. A senior official said Reagan "is not about to shove someone as trusted as Bill Casey out the door at any time -- and particularly not when he is recovering from an operation."Staff writers Bob Woodward and Lou Cannon contributed to this report.