If there is a hell on earth in the United States today, it may be in the maximum-security New Mexico State Penitentiary that stands on a wind-swept desert plain 10 miles south of here.

The pen has been "locked down" in a state of emergency--with recreation programs curtailed and inmates largely confined to their cells and dormitories--for seven weeks since inmates armed with pistols took over a cellblock Aug. 30 and then used knives to slaughter guard Gerald Magee.

The brief takeover was a shocking reminder of the February, 1980, riot when rampaging inmates seized the entire prison for 36 hours, burned it, and executed 33 inmates considered informers, torturing many of them to death with acetylene torches.

In the last 20 months, some of the top professionals in the field have been hired to improve the prison, and a vastly increased budget has been authorized. But still the situation remains violent. Nine inmates and two guards have been murdered since the riots. So many killings have taken place in one exercise yard that only one inmate is allowed to use it at a time, and he is shackled and watched by a nearby guard with a sniper scope rifle who has no other duty.

In the midst of these horrors, a fascinating drama in penology here is being watched closely by corrections officials across the country at a time of growing crisis for the nation's prisons. As the crime rate soars and sentences grow longer, prisons are overflowing, their populations growing at the fastest rate in half a century. Prison conditions can only be described as terrible in most states, and the solutions seem as elusive as ever.

Voters are reluctant to approve funds for new prisons, let alone for supposedly more progressive correctional programs, such as halfway houses. The Reagan administration has ignored a recommendation by its own crime task force for $2 billion in federal prison aid to the states.

Today, the New Mexico pen remains tense and in a virtual state of siege. Interviews with a score of prisoners and staff members during an inside tour last week revealed a world of clanging iron gates and nervous guards rushing about, a world in which men's nerves are stretched to the breaking point.

The place could explode again at any moment.

The rage and bitterness of the inmates and their distrust of the staff is extreme. And in the guards--63 percent of them here less than a year--there seems to be that strange combination of terror and exhilaration felt in any combat zone.

"Puto! whore! " shouted a prisoner from behind a barred window at Deputy Warden Lloyd Mixdorf as he guided a reporter on the tour. Mixdorf turned and said cheerfully, "How are you today?"

At other times, inmates came up to Mixdorf in the halls and politely asked for help on their cases in that deferential, guarded way that characterizes correct, controlled inmate behavior.

Other inmates in a dormitory surrounded a reporter, complaining of guard harassment, inadequate medical service, curtailment of mail and lack of exercise.

Mixdorf said 100 visitors had been inside the prison that day, and the infirmary log showed that roughly 100 inmates had visited it that day.

The chief complaint from inmates was about the attitudes of the guards. Ronnie A. Vigil, dressed in prison denims and doing one to five years for possession of heroin and aiding a felon, said the guards are "snotty. They show no respect whatsoever and they're not respected back....They yell at you instead of reasoning things out: 'I'm giving you an order, a direct order!' "

Mixdorf conceded that because so many staff members are new, many are not properly trained to deal with inmates. He said it takes years for a guard to develop the combination of authority and concern that typifies a good correctional officer.

One of the prison's best guards is Robert A. Conn, a 38-year-old Vietnam veteran who looked sharp but nervous in his starched brown uniform.

Conn said he never experienced the fear in Vietnam combat that he now feels as a correctional officer in charge of 90 inmates in "the hole," the maximum security cellblock within the prison.

He is outnumbered and, like all the guards, carries no weapon. "In actual combat I was on an equal basis. Here I'm not. Here you have to be 100 percent totally alert at all times. You let your guard down and one of these guys is going to stick something in you."

Conn said that trash and water and even glasses of urine are thrown at him and his fellow guards regularly by inmates in the cellblock.

"We are trying to get away from dealing with crises all the time and get to developing programs, but a small number of inmates keep setting us back," said Harvey Winans, brought in as warden last January from Wisconsin, where he had 27 years' experience in one of the nation's most respected corrections systems.

Winans said the lockdown is being eased day by day. His aim, he said, is to create an institution that is "safe, with good programs, good treatment and security.... If you don't have good control and security, you can't have good programs."

But lawyers for the inmates and the American Civil Liberties Union say Winans and the new corrections department chief, Roger W. Crist, are dragging their feet in carrying out hundreds of reforms specified in a court order a year ago.

"It's been a succession of one excuse after another," said Charles Daniels, a lawyer for the inmates. "...It's very depressing. The place is tense. The word 'explosive' is no exaggeration."

By declaring a state of emergency at the prison, officials have automatically suspended--for the time being--the need to comply with the court order. On Oct. 1, Daniels asked that the state and its corrections department be held in contempt for not progressing faster.

"These prisoners are being punished for violating the law, and they have a hard time understanding why the administration shouldn't comply with a federal court order," said Daniels.

Crist, formerly a prison warden in Montana, says the problem is that "we're trying to accomplish everything at one time.... There's just no way that can be done. And there are different ideas on what to do first."

Whatever is done, it is going to cost plenty. After the big riot, a stunned state legislature voted $109 million for new prison construction and boosted the corrections department budget 37 percent.

With the new funds, Crist is adding 372 new guards and other staff members to the payroll, boosting salaries dramatically to attract talent, and doubling the size of the probation and parole department. About $45 million will be spent in the next two years to tear down the existing pen and build a new maximum security facility.

By transferring inmates from the penitentiary to federal and nearby state prisons, Crist and Winans have cut its population from 1,157 at the time of the big riot to about 700 today, leaving a higher guard-inmate ratio and eliminating the double-celling that had added to tensions.

While overcrowding is a problem here, it was not the key factor in the big 1980 riot, according to a special report by the state attorney general.

"The primary factor...was the disruption of incentive controls over inmates. The removal of incentives undermined inmates' self-interest in keeping order and disrupted the nonviolent power sources of convict leaders," the report concluded. "...power among inmates became based more and more on physical violence."

In other words, the prison programs--the sports events, the singing and drama groups and chess clubs--"cannot be viewed as mere window dressing. When programs are used effectively, they are the integral component in a network of control."

According to the report, this network of control deteriorated between l975 and 1980 when political turmoil in the state led to a rapid succession of wardens and corrections chiefs--five of each during that brief period.

Successive wardens tried to keep the lid on with increasingly harsh disciplinary measures, the dismantling of programs inside the institution and the formal development of inmate informers, or "snitches," as a key intelligence tool. Many of these men died in the big riot at the hands of their enraged fellow inmates.

Adding to the tensions in the prison today are ethnic differences--the population is about 55 percent Hispanic, 35 percent white and 10 percent black, with a sprinkling of Indians. Many inmates are formed into ethnically aligned gangs for self-protection.

Finally, the mere fact of the riot itself has led to enormous subsequent pressures on both inmates and staff. Inmates are living with men who may be testifying against them at trials growing out of the riot. Staff members, particularly at high levels, are spending much time testifying and dealing with federal, state and local agencies investigating the prison.

On the tower, Mixdorf, who had 24 years' experience in the Wisconsin system before coming here, greeted guard William A. Tonacchio: "You're doing a 16 today?"

"Either gonna die or get sick," laughed Tonacchio, one of 15 experienced guards working double shifts five days a week to keep the prison fully staffed until more guards are hired.

Mixdorf said the design of the prison, built in the l950s, makes it difficult to control. Half the inmates are in dormitories, but he said all should be in cellblocks.

At least one inmate agreed with this view, saying a cell would give him additional privacy and security.

Another inmate, John Archuleta, even praised the food, calling it "real good. Everything else is cruddy, but the food is real good." Archuleta is doing l6 years for aggravated assault with a firearm and second degree murder with a firearm.

Richard Bickford complained that the prison classification system mixes dangerous with less dangerous criminals.

"All the dorms are maximum and medium and minimum all mixed together," he said. "I'm medium and I have maximums on both sides. It scares me because of the nature of the crime. They're murderers. They got more time. They got less reason to care." Bickford is doing six years for armed robbery.

At the end of the tour, Mixdorf summed up: "Our future depends on luck and how good a job we do. The ACLU sincerely believes--they want to provide a liberal atmosphere. I'm not too far off from them philosophically ; we both believe in fair treatment of human beings.... I consider myself a liberal in corrections matters compared to the public, but the ACLU considers me a punitive type."