Federal Judge Gerald B. Tjoflat warned a Senate subcommittee yesterday that under the federal court system, no procedure exists for a civil commitment to a mental institution for defendants found innocent by reason of insanity.

"There's nothing a judge can do, no way of detaining them," Tjoflat, a judge on the 11th U.S. Circuit Court of Appeals (Atlanta), told a Judiciary subcommittee that is looking into changes in the federal insanity defense.

John W. Hinckley Jr., recently found innocent by reason of insanity in the shooting of President Reagan and three others, was sent to a mental hospital only because the crime occurred in the District of Columbia, which requires a temporary psychiatric confinement after such a finding.

Hinckley could be freed by St. Elizabeth's Hospital here after 50 days if the hospital concludes he is not a threat to himself or others. Tjoflat recommended legislation to empower federal judges to order defendants who are acquitted on an insanity defense to be confined for mental tests.

But mental-health officials said yesterday that there is little evidence that hospitals and psychiatrists can predict future violent behavior in patients in deciding whether to release them.

"We could stand here and flip coins, or a judge could do the same thing, and be as accurate as clinicians who give elaborate presentations of patients based on 30-day studies," said Dr. Henry J. Steadman, head of special projects research at the New York State Department of Mental Hygiene. "There's never been a study that shows a better than 50-50 chance for anyone to predict behavior of someone in a mental facility."

Dr. Stuart B. Silver, Maryland's assistant secretary for forensic services, added, "We cannot predict future acts of violence. We can only make unreliable judgments based on past experience."

Efforts have been under way for some time to change the ease with which defendants can claim an insanity defense, but interest in amending the law mushroomed with the public outcry that followed the Hinckley verdict. Each state also has its own law on insanity defenses.

Most of the publicity is focused on cases involving sensational murders or sex-related crimes that occur after a defendant is acquitted on an insanity plea. But, in fact, the insanity defense has been used in all sorts of cases, ranging from murder to such misdemeanors as vandalism and minor theft.

Steadman cited statistics from a new study that found that 1,625 persons were acquitted by reason of insanity in 1978 and committed to mental hospitals in the United States. He did not break down the type of crimes involved.

Congress is considering changes in the federal law, the most extreme of which would eliminate the insanity defense for any defendant judged competant to stand trial.

Other possibilities include requiring the defense to prove insanity--the government now bears the burden of proof--and increasing the court's jurisdiction over release from mental institutions. This might include probation equal to the time that would have been served with a conviction.