Dr. Phillip J. Resnick had been talking for hours. First, he told his eager charges they would be laughed at by juries and badgered by lawyers. Then he told them to watch out for attorneys looking to trip them up with the "shell game" question, and the "vain Caesar" question.

He told them how to dress. ("Pinstripes enhance authority. Plaid is death.") And he told them how to comport themselves. ("Small men should never carry an umbrella into a courtroom," he said. "It marks you as a wimp.")

Finally, he told the 150 psychiatrists attending his one-day American Psychiatric Association course on how to perform as an expert witness -- that when they marched into an American courtroom they had better be ready to fight like dogs.

"Some of you may be made uneasy by this discussion of packaging yourselves," said Resnick, director of forensic psychiatry at Cleveland's University Hospital. "The reality is the courtroom is an adversary system. If one simply feels the truth will prevail, he is nai ve and he will lose. Attorneys play hardball."

Driven by an increasingly contentious and complicated trial system, expert witnesses of all kinds have become a big growth industry in America. But none plays a more prominent role than does the psychiatrist. For decades, psychiatrists have participated as expert witnesses in American trials, testifying in a wide range of cases. It is the psychiatrist who is called upon to help determine whether a convicted murderer is insane, an accident victim has suffered psychological damage, or a divorced person is a competent parent.

Legal and medical authorities agree that the testimony of psychiatrists can often determine the outcome of even the most public and controversial trials. But they also acknowledge that the credibility of psychiatrists on the witness stand is frequently in doubt.

Part of the problem stems from the nature of psychiatry. When two engineers differ sharply over how a bridge fell down, few people challenge their credentials. When pathologists argued bitterly over the placement of hair tissue in the Scarsdale murder trial of Jean Harris several years ago, close observers never doubted the value of either doctor's training.

But when opposing psychiatrists draw different conclusions from the same evidence, as they often do -- most recently in the Baby M custody trial or the trial of John Hinckley Jr. -- public criticism of psychiatry can become intense. Profession on Trial

Expert witnesses are usually defined as people who have knowledge of facts directly related to some science or profession beyond the scope of the average person one would expect to find on a jury. But like beauty, many of these facts are in the eye of the beholder.

"With something like Baby M or notorious criminal trials, you are dealing with highly personal feelings on the part of the public," said Dr. Robert Sadoff, clinical professor of psychiatry at the University of Pennsylvania. "Everybody feels they have a right to an opinion. These trials become massive morality plays, and, when psychiatrists disagree on the issue of evil or sickness, it looks like they are letting criminals walk away."

Because lawyers can almost always find a psychiatrist to buttress their particular legal position, some suggest these psychiatrists are nothing more than hired guns.

"If I got myself an impartial witness, I'd think I was wasting my money," Melvin Belli, the noted trial lawyer, has said, summing up a common sentiment in the legal community.

There are no fixed fees for psychiatric witnesses. Testimony and preparation for a lengthy trial can earn $10,000 or more for a doctor. Usually, the fees are far lower, and most doctors charge by the hour. Testimony itself is usually only a small fraction of the fee; preparation, interviews and depositions can take much more time.

Just as the range in fees can be dramatic, so can the differences in the quality of the testimony.

"You can't budge the real pros," said Robert F. Horan Jr., for the past 20 years the Fairfax County prosecutor, who has lectured around the country on psychiatric witnesses as experts. "But in the courtroom you tend to see a fairly high percentage of sleazes -- the guys who will put down any diagnosis in the file as long as he gets his check."

Besides, a good lawyer can destroy a bad witness's credibility in minutes. In the 1950 perjury trial of Alger Hiss, the cross-examination of Dr. Carl Binger by prosecutor Thomas Murphy is often cited as one of the most devastating cross-examinations of an expert witness ever conducted.

Hiss, a State Department official, had been accused by Whittaker Chambers of being an active agent of the Communist Party in the 1930s. His defense rested largely on discrediting Chambers' assertions.

Testifying for the defense, Dr. Binger -- who had never examined the accuser -- said nonetheless that Chambers had a psychopathic personality characterized by a tendency to make false accusations. He cited as evidence of Chambers' diseased mind the fact that during his testimony he "frequently looked at the ceiling as if trying to recall something that he had previously said."

Murphy shot back: "Now, Doctor, we made a count this morning of the number of times that you looked at the ceiling, and during the first 10 minutes you looked at the ceiling 19 times, the next 15 minutes 20 times, and the next 15 minutes 10 times . . . I was wondering, Doctor, whether that had the symptoms of a psychopathic personality?"

That and several similar blunders finished Binger as an expert witness on behalf of Hiss and certainly helped convict Hiss.

Less dramatic in the courtroom are the legal tricks of the trade that Resnick warned of in his course on how to be a successful witness.

There is the ploy, for instance, called the "shell game" question. That's when the lawyer attempts to shift dates, places or subjects without advising the witness. The person on the stand is made to appear to confused.

Or lawyers will play on a person's vanity. The "vain Caesar" question, for example, would build the doctor up ("You are a renowned man of science, sir, are you not?") and then suggest he is a fool if he cannot answer a question that nobody would be expected to know. ("Didn't you know that the defendant was likely to kill his mother if he was granted a leave?")

And sometimes the psychiatrists called in to testify are just plain sloppy.

"When I see some of those people, I can pull out a report they did five years ago, just change the names and all the cliche's are just repeated," Horan said.

" I actually did that in court once, and it was devastating. I just read the report and asked if this was what the doctor had concluded," Horan continued. "When he said yes, I showed him and the court that the document was five years old -- and about another person. After that, if he had said 'light comes from the sun,' no jury in America would have believed him." Scientific Gray Zone

The credibility issue is rooted in the fact that unlike the "hard" specialties in medicine, such as cardiology or oncology, psychiatry is a complicated discipline that mixes art and science. That means honest answers given by psychiatrists on the witness stand can cause them their biggest problems.

"The public tends to view general medicine as a reasonable science," said Dr. Mortin Orne, who has testified in many famous trials, including those of the Hillside Strangler, who killed a series of West Coast women, and Patty Hearst, after she was kidnapped by the Symbionese Liberation Army. "But psychiatry deals with issues that are almost political. Two psychiatrists may look at the same observations and agree but interpret them in fundamentally different ways. It is just not a pure science. It is in large part still an art with scientific components to it."

It is the art, no matter how sophisticated or subtle, that appears to infuriate people. Nowhere is the public debate more heated than over the insanity defense.

"We tend to say somebody is crazy if we have no way of understanding his motivation," said Orne. "But that causes some special problems in the courtroom."

Orne and other psychiatrists agree, for example, that there is one type of serial killer who can never get away with an insanity defense.

"That's the hit man," he said. "And that's very inter- esting, because many who have been hit men are ex- tremely psychotic. But no jury will agree because they say he took money to kill and therefore it makes sense."

Yet if it is a murder with no apparent motive, the same

diagnosis of a psychosis may be viewed differently by a jury. "If I say he is killing because he talked to the moon and the moon told him to shoot some guy, people would believe he was crazy," Orne said.

In general, legal officials say, courts are looking at the insanity plea with a cooler eye.

Psychiatrists examining Hinckley testified that he was insane and could not be held criminally responsible for shooting President Reagan and three others in 1981. After he was found not guilty on that basis and committed to a mental institution, the resulting furor caused many states to make it more difficult for defendants to use the insanity defense.

Earlier this month, a Fairfax man who tried to invoke the insanity defense at his trial was executed. Richard Lee Witley, a part-time house painter, was convicted of the brutal murder of a 63-year-old woman who lived next door to him. His attorney based several appeals on Whitley's IQ, which tested at 75, saying he was "feebleminded" and "insane." The crime was particularly gory, however, and neither the courts nor Virginia Gov. Gerald L. Baliles would overturn the 1981 decision.

Professional opinion on sanity, or competence, almost always has shades of gray that cannot be simply resolved. The criteria governing sanity or competence are rarely clear. Most of the time, communicating that subtlety to the jury is the central goal of psychiatric testimony. But the nature of the legal system is to pressure witnesses to go beyond their expertise.

Clinicians, for example, are often asked to testify about what a person's state of mind was six months or a year ago. Most psychiatrists agree this is a very difficult position to have to take.

Often, they are also asked to predict the future behavior of defendants who have committed crimes of violence in the past.

"Psychiatrists have no better ability to guess what someone will do in the future than does a third-grade teacher," says Resnick, one of the country's most noted forensic psychiatrists. "But you can find all sorts of people who will tell the jury how certain they are that a man will kill again."

Predicting future dangerousness is something that the American Psychiatric Association has vigorously attacked. But many prosecutors are prepared to scour the country for a doctor who will tell a jury that the accused will repeat their violent crimes. Usually, the search is easy.

Take Dallas psychiatrist Dr. James Grigson, for example. Over the past decade, he has made a profession of testifying for prosecutors seeking the death penalty. In dozens of cases, he has predicted that the criminal defendants would, without question, be dangerous in the future.

In most cases, Grigson spends no more than 90 minutes with the person before he predicts their future behavior.

But juries respond to his certainty. In the majority of cases in which he has testified, juries imposed the death penalty. Irreconcilable Differences

Both the APA and the American Bar Association recommend that psychiatrists lim- it their testimony to those specific areas where they have a real expertise. They are supposed to leave the conclusions to the court. But there is always an ultimate legal issue, and experts are pressed hard to make a judgment. It makes the job easier for the judge and jury.

"The rules are ignored constantly," said Dr. Paul Appelbaum, professor of psychiatry at the University of Massachusetts Medical School. "If you play it by the book, many lawyers just won't hire you. They feel you are less effective if you are not willing to say, 'and therefore he should be found not guilty.' "

Many legal professionals, and some psychiatrists, now feel that the worlds of law and psychiatry are so distinct that they can never truly be reconciled. Some have suggested that psychiatric testimony be admitted only after a verdict has been reached. That way, the testimony could not influence decisions on guilt and innocence but simply on the terms and place of incarceration if a defendant were found guilty.

Others say that increasingly juries need the opportunity to ask questions of experts, particularly during long and contradictory testimony. Currently, juries are not even permitted to take notes during testimony. "I don't think you can seriously argue that psychiatrists should be totally eliminated from the courtroom," said Roger Adelman, Assistant U.S. Attorney for the District of Columbia, who prosecuted Hinckley. "But we need to overhaul the process so that judge and jury can ask questions. We need interim summations and more interaction. Doctors come from a world where their word is taken as a commandment. That doesn't work in court."

Like many others, Adelman notes that there is a real incompatibility between the psychiatric and legal world views.

"In law, we ask for a definite conclusion about a person's mind at a specific time," he said. "Medicine can't do that. We talk about malice, intent, etc. This language makes psychiatrists uncomfortable. And they uniformly resent the interrogating aspect of the adversary process."

As a result, many of the best mental health professionals avoid appearing in court completely. They don't believe that the adversary system and medicine can mix properly.

Yet experts clearly have a role of growing importance to play in complicated and even emotional decisions. Juries often need sophisticated, technical information that only a trained professional could provide.

"The most important thing is that the mental health professional must never make decisions in court," said Lee Salk, the Manhattan psychologist who testified on behalf of William and Elizabeth Stern in their successful custody battle with Mary Beth Whitehead, who bore the child, known as Baby M, for the couple. At the trial, Salk testified that Whitehead, the mother of Baby M, had a "severe personality disorder," emotional "instability" and a "serious deficit in her capacity to parent."

He was paid $5,000 for his testimony as an expert witness by Mr. Stern.

The Baby M case made history primarily because it was the first in the nation to test a surrogate mother's rights. Throughout the lengthy trial, a parade of experts contradicted each other in the vague new arena that many in the public followed with acute interest. "We don't have an established mothering meter, so there was a lot of doubt and anger," said Salk. "No matter what you said, you were going to be attacked for it."

Cases that deal with delicate human behavior, where nothing is black and white, are often the most controversial. "Appearing in court over something like this is a bad situation but it has to be done," Salk said. "I would have prefered not to testify. But I did have an opinion, and I came by it honestly."

Psychiatric testimony in family law cases -- particularly where custody of children is at stake -- has increased dramatically in the last few years. While the issues are not at all similar to those in insanity cases, they are equally difficult to resolve. Custody trials, for example, usually revolve around who is the more competent parent -- a psychological gray zone where psychiatrists are often asked to make a value judgment about personality rather than a diagnosis of a mental disorder.

Psychiatrists argue that the cases that come to trial are in no way representative of their work or the interaction between the legal system and medicine. Indeed, the vast majority of lawsuits and criminal cases never do come to trial. In medical malpractice cases -- where expert testimony is vital -- fewer than 10 percent of all cases end up in court.

"The cases that actually do come to trial are always most likely to be in that gray zone where neither side is clearly in the right," said Dr. Howard V. Zonana, director of the Psychiatry and Law Program at Yale University and chairman of the APA council on Law and Psychiatry. "The ones you see and read about are those where a difference of opinion is most likely to arise," he said.

"Most of the time, conflicting testimony is not an indication that psychiatrists cannot agree with each other on anything. It means they are dealing with a very troublesome issue."

Zonana and some of his colleagues warn that among the biggest pitfalls for psychiatrists is the appearance that they are too highly identified with the side that hires him or her.

"It is a courtship," said Zonana, "a process of seduction between an attorney looking for an expert and the psychiatrist. The lawyer courts the expert. He is nice and he is flattering. Over time those factors, more than who is paying the bill, make it hard to say, 'I'm sorry, Joe, but there isn't a thing to support your case here.' "

Most forensic experts agree that psychiatrists must, as best they can, keep a personal distance from the lawyers they represent. In his course, Resnick tells of a psychiatrist who upon finishing his testimony flashed a victory sign to the lawyer who hired him. "It was all over immediately," said Resnick. "That kind of arrogance makes a jury sick. When you are done with your testimony, stand up and leave the courtroom."

Despite questions of professional ethics, psychiatrists are clearly in the courtroom to stay. They are needed particularly in complicated cases where jurors need a sense of what the range of accepted behavior can be, whether it's a custody dispute or an occupational injury.

"Look, we play an important role," said Zonana, who has recently examined several survivors of a building collapse in Bridgeport, Conn., that killed 28 construction workers on April 23.

"What do you do with a man who watched his son die right in front of him? He was not hurt physically, and the Workers Compensation people look at him and say, 'He looks all right to me.'

"Well, the man doesn't sleep. He has nightmares. Those are real, tangible consequences of that disaster. We know all about how that happens. But if he is going to get a penny in workers' comp, he is going to have to get it in a courtroom."