The mentally ill are coming home.

While they once spent years behind hospital bars as relatives made excuses that Aunt Mary had moved out West or brother John had joined the Navy, they now often spend several weeks in a psychiatric hospital, are released and then return as their condition warrants. (Eight out of 1,000 Americans were committed to a hospital for psychiatric care in 1979, according to the latest statistics available from the National Institutes of Mental Health.) The "revolving-door" model of treatment is a result of legal time limits on institutional confinement and new drugs that relieve many symptoms.

Although the more flexible arrangement may be desirable for some of the mentally ill and their families, it can create a painful no-win bind for families who have not bargained for the life-long dependency of one of their members.

Finding after-care services for these returning family members is "driving families up the wall," claims Phyllis Vine, author of Families in Pain (Pantheon, 273 pages, $15), a handbook for relatives of the mentally ill.

"Families need help in finding a place for their mentally ill members to live and work. But often support is limited to family therapy, which focuses on how the family contributed to the problem, rather than how they can jointly solve the problem, adds Vine, a history professor at Sarah Lawrence College, who wrote her book after realizing that her struggle to help a mentally ill cousin was "a social problem and not just a personal problem."

"Families also want answers," says Vine, "but they may be kept in the dark for weeks with no explanation, because a lot of diagnosis rests on seeing what medication works."

Diagnosis for 19-year-old Beth is as elusive as the field of psychiatry itself. In the seven years since Jan first saw her daughter "fall apart," she has been labeled schizophrenic, learning disabled, mentally retarded and epileptic.

"If Beth were clearly one thing or another, it would have been easier," says her mother. "But the diagnosis has shifted so much, I haven't had a reliable guidepost to go by."

An acute panic attack in a seventh-grade science class was Beth's first major symptom. A school counselor advised Jan to take Beth out of school "before the wolf packs form."

Beth spent months at home eating and sleeping poorly, because she was terrified she was going to die. "She wandered through the apartment all night mumbling to herself," says Jan.

"After watching her 24 hours a day, I became a basket case. So I placed her in a children's hospital that specializes in learning disabilities. She improved and was able to return to public school, but then she became unglued again two years ago, and it's been pretty bad ever since."

Jan's next strategy is to place Beth in a residential program that stresses training in social and vocational skills.

"I hope she will agree to this," says Jan. "If not, I don't know what I will do. She is so angry at me, because her life doesn't work. And that's what mothers are for . . . to make life work."

"What is so difficult about mental illness is that it is unpredictable," says Vine. "With a physical handicap you learn to adjust, because you know what has to be done. But with mental illness you can't plan on how to meet someone's needs, because you never know what they will be and when they will emerge."

Feelings of hopelessness can be exacerbated by the fear that other relatives also will become ill. "This can become an obsession," says Vine, "in which people look for psychotic symptoms and stop enjoying life. It also adds tremendous guilt in which the healthy members ask, 'Why didn't I get it?' "

As she sat in a psychiatrist's office after her second sister's commitment to a psychiatric hospital, the foremost question in Kathy's mind was, " 'Am I next?'

"He assured me that mental illness is not hereditary, and I guess I am the survivor of the family. Growing up was hard for all of us, and I had a time in college when I was very depressed, but I've never gone crazy."

Joan, the eldest and her mother's pet, was a straight "A" student when her dorm mates found her lying on the floor in a position of crucifixion. A sign on the door proclaimed, "I am the Messiah."

When Joan was released from a psychiatric hospital, her parents punished her for her breakdown by ignoring her. She turned to her college boyfriend for support, married and isolated her new family in a small Christian community.

Labeling Joan an "emotional cripple," Kathy feels confused and impatient with a sister who won't take a family trip to the zoo because it will interfere with her rigid schedule of naps and every-half-hour health-food snacks.

It is even more painful for Kathy to accept the mental illness of a younger sister--Nancy--whom she comforted for years as a "surrogate mother." When Nancy was 19 years old, she called Kathy and "was really freaked out. She said she had just painted a beautiful picture that was all black with a little white bird, and she didn't remember doing it." Soon after, she tried to kill herself with an overdose of aspirin and had to be hospitalized.

Although both sisters seem to be improving, Kathy says she is afraid to get her hopes up. "If they break down again, I will be upset. I can't help feeling it is something they can control."

Pointing to the mentally ill as somehow to blame for their condition is a common reaction, despite growing evidence of the biochemical origins of many psychological disorders.

In her interviews, Vine found that both patients and their families are often punished for their "deviancy" by their communities.

"People don't want to come over and visit after they discover someone in the family is mentally ill," says Vine. "The patient becomes a nonperson and may even see his name taken off the church roster. But just because he has this problem doesn't mean he should be treated as if he is socially dead."

Cindy, 36, grew up almost believing her family life was normal because it was familiar. But she was also frightened by her mother's erratic behavior.

When Cindy was a preschooler, her mother Mary was "taken away" and given electroshock therapy to curb her hallucinations. Eight years later, when Mary saw the Virgin Mary and Wise Men congregating by a grocery-store checkout counter, she sought psychiatric help again, only to be scared away by the threat of a second hospitalization.

"For us the prospect of locking my mother away in a mental ward was more frightening than foregoing her treatment," says Cindy. "We rationalized that psychiatrists were just out to get you, and that my mother was a special person, because she was having religious visions.

"We were also afraid of the social stigma that our family was somehow bad or disgusting, because my mother was mentally ill. So we protected her and ourselves by pretending that everything was okay. She didn't have any friends and was a hand-wringing worrier, but, apart from her religious experiences, she really didn't seem too different.

"When I see how things are today, with short-term hospitalization and the different kinds of after-care, I regret that it wasn't available for her. As it was, we were faced with an all-or-nothing proposition, and no one wanted to send her off to some snake pit."