The controversial "insanity defense" under which John W. Hinckley was found not guilty of the attempted assassination of President Reagan should be tightened so persons with antisocial personality disorders are not released into society after committing crimes, the American Psychiatric Association said yesterday.

The group, which is the major psychiatric organization in the United States, called for new steps to protect the public from the release of both "antisocial" and "insane" persons who have committed crimes.

Psychiatrists are unable to predict which persons might be dangerous, the group said, and their testimony in court cases should be limited to the area of their expertise--the defendant's mental state and motivation.

At the same time, the APA defended the insanity defense because it rests on a fundamental tenet of criminal law, "that punishment for wrongful deeds should be predicated upon moral responsibility"--that persons so mentally confused that they don't comprehend what they are doing should not be expected to bear responsibility for something they can't understand.

The APA, which represents 28,000 American psychiatrists, urged four far-reaching changes in the handling of defendants who plead insanity:

* Acquitting persons on grounds of insanity only if they if they are diagnosed as having a "serious" mental disorder, which usually means a psychosis rather than the less-serious "antisocial personality disorder."

A psychotic can be defined as someone who is frequently out of touch with reality. A person with an antisocial personality disorder usually knows he is committing antisocial acts but doesn't care.

* Holding persons with antisocial personality disorders "accountable for their behavior," in accord with latest psychiatric thinking and treatment.

Psychiatrists believe antisocial persons should be forced to face the consequences of their acts.

* Asking psychiatrists to testify only on their area of competence--the nature of a defendant's mental state and motivation--but not on whether a defendant is "insane" or can be held legally "responsible."

These are legal and social questions on which only a jury or judge are "the experts," the APA said.

* Freeing possibly dangerous persons who are judged "not guilty by reason of insanity" only after a body such as a parole board determines the best course to protect the public, as psychiatrists themselves are unable to predict "dangerousness."

In some cases, the APA said, this might mean conditional release; in others, compulsory treatment, and sometimes confinement in a mental hospital or, if no hospital is appropriate, "a non-treatment facility that can provide the necessary security," which could mean a jail or prison.

The public, the APA said, is not being well served by releasing such persons.

"Even a man who has killed may receive only 50 days of confinement in a mental hospital," said Dr. Loren Roth, director of law and psychiatry at Pittsburgh's Western Psychiatric Institute, at a news conference.

Roth chaired an Insanity Defense Work Group named by the APA last year in the wake of the furor over the Hinckley verdict and new debates on the insanity defense in Congress and several state legislatures.

The work group's report, adopted by the APA trustees, is the first comprehensive one on the subject in the APA's 138-year history.

Of Hinckley, Roth would say only, "I would not presume to diagnose him since I did not examine him."