Deinstitutionalization, the policy conceived in the optimism of the 1950s, is being reevaluated in the disillusionment of the 1980s.

The burgeoning number of mentally ill street people has prompted some psychiatrists and politicians to advocate the establishment of involuntary "asylums" to prevent ex-patients, in the words of one doctor, from "dying with their rights on."

But most mental health experts and policymakers say they still believe treatment in the community is more humane, more therapeutic and less stigmatizing than confinement in a state hospital.

"Is deinstitutionalization a failure? I say we won't know until we try," said Leona L. Bachrach, an influential sociologist at the University of Maryland and a consultant to the American Psychiatric Association. "I see it as a failure not of the policy but of the implementation."

Nonetheless, the conspicuous inability of most states to care for former patients has resulted in a reexamination by some psychiatrists of the basic assumptions behind deinstitutionalization.

"A tremendous error was made on false premises, which is that patients become chronic because they were in a hospital and that state hospitals were snake pits," said Alexander Gralnick, a Westchester, N.Y., psychiatrist who wrote "The Case Against Deinstitutionalization," published in a recent APA journal. "The best place to treat mental illness is in a hospital because we know so little about what started it and what perpetuated it."

A similar view is echoed by Dr. Stephen L. Rachlin, chairman of the Department of Psychiatry and Psychology at Nassau County Medical Center in New York. "Let's stop playing silly games based on philosophical slogans," said Rachlin, who calls himself "a medical chauvinist pig."

"There are people who are going to need current technology, and they need a place called a hospital where they can stay," he said. Many chronically mentally ill people, Rachlin said, "are not very pretty, they're not fun to be with . . . . But who the hell is going to take care of these people if not the state hospital?"

Some of the architects of deinstitutionalization admit that the policy was oversold on the false promise that states could save a lot of money by treating people outside hospitals. They could -- chiefly by discharging patients and not providing community programs.

"In the typical community and for the average patient deinstitutionalization has been a failure," said Joseph P. Morrissey, a research scientist with the New York State Department of Mental Hygiene. "There has been a lot of motion but little progress."

Although the concept of asylum has gained increasing currency, it is still largely undefined. Its proponents, who include APA President John A. Talbott, envision them as small, modern sanctuaries that protect the very ill -- not the giant institutions built a century ago.

Civil libertarians say that while the term conjures up images of bucolic rest homes, it really means refilling underused mental hospitals.

"I think it just means reinstitutionalization," said Norman S. Rosenberg, executive director of the Mental Health Law Project. "Some people do need to be institutionalized . . . but you are essentially talking about a life sentence for a lot of people."

Virtually no one thinks big state hospitals will be refilled, if only because the cost is prohibitive. Federal and state governments are looking for ways to cut costs, not spend more money. In 1980 the United States spent $30 billion on mental health services, much of it on institutional care.

Dr. Bertram S. Brown, former director of the National Institute of Mental Health, said he thinks a final judgment about the policy he helped conceive is premature.

"Society is at a crossroads, as the half-dozen of us there at the genesis knew we'd be," said Brown, now president of Philadelphia's Hahnemann University. "Communities have to decide whether to finish the job of adequate community support or go back to the asylum: out of sight, out of mind. My sense is that it'll be a split decision."