THE AMERICAN PRISON system, as everyone knows, is in deep trouble. What is less recognized is that prison authorities are not merely losing control over prisoners -- they are losing control over their own personnel.

In the first five months of this year, major prison disturbances were reported in New Mexico, Louisiana, Maryland, Indiana, Nevada, Michigan, Illinois, Washington state, Hawaii and New York. One of the most serious occurred on May 22 at Michigan's Jackson State Penitentiary, largest walled prison in the world, holding pen of 5,000 convicts. It was quickly followed by riots in two other Michigan institutions.

One of the most worrisome aspects of the JAckson episode was that it was triggered, according to state officials, by unauthorized actions by the prison's own guards. After correction officers were refused permission to conduct an institution-wide lockup and shakedown, they went ahead and did it anyway, defying orders by Warden Barry Mintzes and State Corrections Director Perry Johnson. The convicts resisted and a violent confrontation -- resulting in destruction estimated at over $5 million -- ensued.

Only last year, one of the bloodiest riots in American prison history erupted at the New Mexico State Penitentiary, where 33 inmates were stabbed, beaten or tortured to death by fellow convicts during a 36-hour orgy of violence. The savagery was so intense that it tended to obscure many of the factors which had brought it about.

Since the early 1970s, guard turnover at the penitentiary had risen from about 30 percent to nearly 80 percent, indicating that guards as well as prisoners were finding conditions intolerable. Repeated reports since 1975 had stressed an almost complete breakdown in security, lack of proper supervision and guard misconduct. Only three months before the bloodbath, the escape of 11 prisoners had resulted in the dismissal of two guards for negligence.

In the wake of several recent disturbances, some prison officials have begun to take action against their employes for improper conduct leading to or following riots. Three weeks after the Jackson incident, the Department of Corrections dismissed a deputy warden and two guards, suspended 14 guards, and demoted two prison administrators. Five more employes are under investigation and could face disciplinary action, according to director Johnson.

The dangerous tendency of some correction officers to take the law into their own hands was graphically revealed last year, when a court-appointed monitor in Georgia reported that guards at Reidsville State Prison had imposed a virtual "reign of terror" against inmates. According to documents filed in Federal District Court in Atlanta:

"All staff, including some high-ranking administrators, as well as inmates, have acknowledged that for a period of at least several months following the riot, extensive misuse of force occurred on a daily basis."

In New York, a state anticorruption commission this spring issued a scathing report documenting widespread and systematic corruption among employes at Green Haven State Prison, a maximum-security fortress containing some of the state's most dangerous criminals. Among its findings:

Guards received payments in cash, stolen goods and sexual favors, in exchange for allowing "prisoners with organized crime connections" to make unauthorized trips "to stop for costly meals, to visit with criminalassociates, to have sex or to walk off unattended." Two officers allegedly allowed at least one notorious prisoner to escape, and then covered it up. The chairman of the agency which conducted the probe has cautioned that such rampant misconduct is not unique to Green Haven, saying: "We need a look at the whole system."

There is near-unanimous agreement that American prisons are severely overcrowded starved of adequate resources for rehabilitating offenders or deterring crime. With more than 300,000 persons in state prisons, and another 200,000 in local jails, the United States has more of its citizens behind bars than any other nation in the Western world --signed to hold.

Increased demands on the prison dollar have produced deteriorating conditions for personnel and prisoners alike.

Control of a prison used to be concentrated and hierarchical, according to a power structure modeled along military lines. But the old Hollywood image of a benign or autocratic warden presiding over his well disciplined troops has become obsolete in most settings. Many wardens today are little more than figureheads, and their superiors -- the commissioners of correction -- probably enjoy the briefest tenure of any state cabinet officers. The institutions are the object of intense power struggles, involving a host of opposing interests.

Correction officers in many system have become organized into vocal, often militant, labor unions. A two-volume study of guard unions, funded by the federal Law Enforcement Assistance Administration, has concluded that this new development in corrections has greatly increased guard salaries and power -- but impaired many prison reform efforts.

In some states it is the guards -- not the administration -- who come closest to running the institutions, sometimes by intimidating inmates and higher authorities alike. Officers and administrators regard each other as "management" or "labor" adversaries. And while prisoners and staff alike demand to be treated as human beings, some find it hard to accord the other the same right.

Amid this confused, often heated, welter of antogonisms, there are many chains of command and many conflicting loyalties and objectives. Whom does one follow: commissioner, administration, union, gang leader, partner or some other authority?

Many guards are as hostile as the convicts toward "the system" and nearly everyone feels that he or she is not getting a fair shake. It is difficult to disagree: Prisons are unpleasant places. About as many guards quit as prisoners return, for obvious reasons, and the routine they endure is hardly designed to engender trust or good will toward the other. To a large extent, the members of each group conform to the same code: Don't rat on your brother, cover your tail and protect your own interests. Nobody accepts responsibility for what goes down.

Following a riot at Washington's Walla Walla State Penitentiary, a blue-ribbon commission offered the following comment:

"There is an almost unbelievable lack of communication between the administration, the commond staff, the correctional officers, and vice versa. A second serious breakdown is in the chain of command . . . Sergeants do not seem to have any authority as sergeants. Lieutenants seem to be afraid of, or unwilling to accept, authority, generally displaying an attitude of 'handle it any way you have to; don't let me know how you do it, and write a clean report when you're through.'"

At Walla Walla, "handle it any way you have to" meant permission to resort to beatings, destruction of inmate property, punitive cell shakedowns and a variety of other abuses. At this point, any attempt by management to intervene and stop the excesses would have been tantamount to an admission that managerial supervision had not taken place. It seemed safer to pretend that the abuses had never occurred. The guards had been given a free hand to commit illegal acts, And management became, in effect, hostages of their employes.

Prison officials can't expect prisoners to correct themselves unless the authorities first clean up their own act. They can begin by conforming to standards of law and decency. Until this happens, the institutions they keep are bound to remain mean and incorrigible monuments to lawlessness and disorder.