THE HOUSE Judiciary Committee has under study proposals to give the Department of Justice power to initiate legal actions to protect the rights of inmates in state institutions. The proposals are controversial and not free from constitutional doubt. The National Association of Attorneys General has denounced them as invasions of states' rights and promised to contest in court any that is enacted. But it seems to us that the need for legislation in this field is clear and that the legal basis for it is sufficiently strong for Congress to adopt one of these proposals.

The problem at which the legislation is aimed is the way inmates are treated in some state prisons, hospitals and mental facilities. The evidence compiled in a series of recent court cases has been staggering. In states as diverse as Alabama and New York, federal judges have ordered sweeping changes in the way institutions are run after finding substantial evidence that inmates were systematically abused and denied their constitutional rights. The mistreatment that was documented ranged from failing to protect convicts from physical assaults by other convicts to ignoring the medical needs of the mentally ill to simple malnutrition.

Most of these court cases were brought by individually inmates, and the Department of Justice was permitted or asked to intervene by the trial judge. But the experience gained in them convinced the Department that there are a number of other state-operated institutions in which maltreatment occurs but in which no immate is able or dares to go into court. It filed two cases itself - one in Maryland over conditions at Rosewood state hospital and one in Montana - but both were dismissed on the ground that the executive branch of the federal government lacks authority to file such a case without a specific grant of power from Congress. The proposals now under consideration in the House Judiciary Committee would provide that authority. They have the support of the Carter administration and the American Bar Association.

The states, however, have been attacking the proposals as unconstitutional, unwarranted and unwise. The Association of Attorneys General argues that principles of federalism bar the federal government from interfering so deeply in matters entrusted to the states. And they contend that solutions to problems can be worked out better through cooperation than through law suits that put the problems in the hands of federal judges who are poorly equipped to tell state governments how to run particular institutions.

The trouble with these arguments is that too many states have shown a disinclination to clean up the messes in their own institutions until the federal courts have ordered them to do so.It would be much better to keep matters like this out of the courts, but the issue is not unlike that of reappointment: History demonstrated that unless the courts began to require it, some states would never fulfill their constitutional obligations. We have seen precious little to suggest that the states will begin to treat all inmates the way the federal constitution requires unless they are pushed into it. And the best way for that pressure to be brought, it seems to us, in a coordinated program run by the Department of Justice rather than in a long series of individual law suits filed by inmates who have been abused.