The battle of the experts intensified yesterday in the case of accused IBM slayer Edward Thomas Mann, with psychiatrists strenuously disagreeing on whether he is mentally competent to stand trial and one doctor conceding that "the case is too close to call."

In hearings that began last month, Montgomery County Circuit Court Judge William C. Miller is attempting to determine whether Mann, accused of murdering three persons in a rampage at IBM's Bethesda headquarters last May, is rational and able to stand trial.

Yesterday's hearing ended when Dr. Leonard Hertzberg testified, "The case is too close to call . . . . I see this as borderline type case. I see Mr. Mann as a very ill man . . . and he is not competent in my mind beyond a reasonable doubt."

Defense attorneys maintain that Mann is incompetent. In order to place him on trial, the prosecutors must prove beyond a reasonable doubt that Mann meets the legal test for competency--that he can understand the legal proceedings and assist in his own defense.

So far, although all the experts agree that Mann is suffering from paranoia, they disagree on its severity and their findings on the competency issue have been conflicting. In earlier testimony, three psychiatrists called by the prosecutors declared that Mann is competent and four experts called by the defense said he is not. Yesterday's testimony added one more voice on either side.

Dr. Michael Spodak, of the Clifton T. Perkins State Hospital where Mann was evaluated, testified for the prosecution that although Mann suffers from paranoia, "it has not impaired his functioning. He provides logical, reasonable explanations for his decisions."

Among these decisions were Mann's dismissal of the prominent Washington attorney who represented him, and his requests to defend himself and to replace his plea of not guilty by reason of insanity with a guilty plea. Judge Miller has refused to accept his requests until he can determine Mann's competency, which he is expected to do after two more days of hearings.

When the hearing opened last month, the experts testified that Mann is suffering from the mental disorder of paranoia--the feeling that his former employer, IBM, is conspiring to ruin his life. But they could not agree on the crucial point, whether the paranoia is so severe that it impairs his ability to understand the trial proceedings and assist in his own defense.

Hertzberg, also from Perkins but called by the defense, weighed in yesterday on the side that Mann's paranoia is so severe that it affects his ability to assist his attorneys.

During yesterday's proceedings, the 38-year-old defendant was generally calm and polite, except for one point in Spodak's testimony. Previously some defense experts have testified that Mann is pleading guilty in order to make a statement about the supposed IBM plot against him. Spodak testified yesterday that Mann had told him that he "in some vague way wants to make a statement," but that Mann never elaborated.

To that, Mann shouted, "It's nobody's damn business until I make the statement." Miller replied, "It is the court's business," and Mann, in his customary way, thanked the judge and then fell quiet.