At the end of a 5 1/2-hour session that concluded at 1:30 a.m., a Virginia House committee today killed a controversial proposal to rewrite the law governing the involuntary commitment of persons to the state's mental hospitals.

The surprise 12-to-5 defeat came despite comments by several of the bill's critics that, if modified, the measure would prevent the apparently widespread abuses in the current commitment procedures. The legislators said, however, that the 14-page bill was too complex and included too many disputed provisions to be considered when it finally was brought up for debate.

"It was five after one when it came up," said Del. Jay W. DeBoer (D-Petersburg). "Nobody was physically or mentally alert enough to draft the amendments that needed to be made."

Equally important, the bill's chief sponsor, Del. Warren G. Stambaugh (D-Arlington), for the first time estimated that the measure would cost the state between $6 million and $22 million over a two-year period. The size and range of the figures, computed by state mental health officials, appeared to stun some members of the committee, some of those present said.

"If you can't estimate the cost of a program any better than that, you don't know what you're doing," said Del. Frank M. Slayton (D-Halifax).

The bill grew out of a year-long legislative study subcommittee, chaired by Stambaugh, that identified numerous civil liberties abuses in the procedures by which Virginia courts commit persons to psychiatric hospitals. According to one study by the state mental health department, one third of the approximately 6,000 persons involuntarily committed to institutions every year may be confined illegally--a situation the president of the American Civil Liberties Union has called "scandalous."

The new legislation provided increased legal protection for persons faced with commitment petitions and tightened the two criteria--dangerous to others and inability to care for oneself--under which judges can now order commitments. But the bill also added a new criterion, proposed by the American Psychiatric Association, that would have allowed commitments for persons found to be "likely to suffer substantial mental or emotional deterioration."

That language stirred heated opposition from the ACLU, whose officials said they feared it was too vague and so broad that it would lead to substantially more commitments. Had that and several other provisions been improved, the ACLU would have supported the bill, state ACLU director Chan Kendrick said today.

"I'm clearly disappointed," Kendrick said today. "A revision in the law is desperately needed . . . But I don't think it's needed enough to put that loose and broad a standard into the law."

A tired and clearly annoyed Stambaugh said today that he would have accepted amendments deleting the controversial provision had one been offered, but that he didn't offer one himself because "I'm not going to amend my own bill." The Arlington Democrat also said that, despite the interest of some committee members today, he doubted he would support an effort to revive the measure before Sunday's deadline for House committee action on House bills.

"I can't deal with every single damn problem," he said. "I spent a year trying to correct a situation where many of the people are being committed illegally and then we got bogged down by picayune details."