Some of the jurors in John W. Hinckley Jr.'s trial yesterday expressed misgivings about their verdict of not guilty by reason of insanity and said they believe he should first be treated in a mental institution and then serve a prison term.

Five of the 12 jurors appeared voluntarily before a Senate Judiciary subcommittee and said they were frustrated by a system that gave them only two choices--guilty or not guilty by reason of insanity. All agreed that they would have preferred an option of finding Hinckley guilty but insane.

In the trial, U.S. District Court Judge Barrington D. Parker ruled that the prosecution had to prove beyond a reasonable doubt that Hinckley was sane on March 30, 1981, when he shot President Reagan and three others outside the Washington Hilton Hotel.

"Everyone knew beyond a shadow of a doubt that he did it. He was guilty. But we had that mental problem to deal with," said juror Maryland T. Copelin of Southeast. "We couldn't do any better than we did on account of the forms" that require the jury to find the defendant either guilty or not guilty.

Copelin said she believes "if the person was guilty of the crime and mentally ill, they should be treated for the illness . . . and after that let them get the punishment for the crime they committed."

Several bills are pending before Congress to change the insanity defense. Some of the legislation would also address the burden now on the prosecution in federal cases to prove that a defendant is sane. There is widespread sentiment in Congress to change the law so that the defense would have to prove that a defendant was insane.

Juror Nathalia L. Brown of Northeast said: "I feel the prosecution did a good job. They went all-out. . . . But putting the burden on them to prove he is sane is hard. . . . We all have some sort of mental illness. We all go through some stress, strain, depression."

The jurors indicated that they felt somewhat uneasy about the unlikely possibility that Hinckley could be free after only 50 days if the court determines he no longer is a danger to himself or others.

Brown said: "I don't believe some jurors knew he had a chance of getting out if he could prove he would not harm himself or others. I had some doubt myself about what would really happen to him if he was found insane."

Juror Woodrow Johnson of Southeast added that if Hinckley is released quickly, "I'd feel guilty."

Some of the jurors also expressed misgivings about the hours of complex psychiatric testimony in the eight-week trial. They said the testimony was difficult to understand and that they thought the defense psychiatrists had been manipulated by Hinckley.

Brown complained that the psychiatrists gave long, confusing answers. "By the time they get around to telling you what it's about, you forget what the question was," she said.

Brown believed Hinckley was guilty and held out for that verdict until the last minute. "He contradicted himself so much. He made fools out of a lot of psychiatrists," she said. "The defense psychiatrists didn't know their job. Hinckley really manipulated them. I don't see how anybody really could have gone too much on their testimony."

She said she tried to convince the other jurors that Hinckley was not insane, but "the prosecution didn't do it. How was I going to do it?"

"I figure if you are insane to this degree to shoot the president , you'd still be insane afterward during the trial . But the way it appeared to me, he had all his faculties then and even during the trial. He had some mental disorder, but that wasn't to the degree that he didn't know what was going on. . . . I feel he knew what he was doing."

She added, " 'Insane' is a word that is hard to figure out. . . . We can say insane and think of somebody being crazy. But legal insanity. How far does it go? We really don't know."

Copelin said she, too, was confused and that the psychiatrists, "with all their knowledge and degrees, they didn't prove anything. . . . You know a person has a disturbance, but you can't say to what degree--not a layman, not me."

Johnson also said he didn't believe that any of the experts had unraveled what was going on in Hinckley's mind the day of the shootings. "Nobody knew what was in the head of the defendant that day," he said.