It was 1:30 a.m. on Jan. 6, and the dog was suddenly barking at someone coming over the convent walls.

It was the moment of terror that haunts so many people in Guatemala, the moment that has multiplied into the tens of thousands during the past 20 years, that has made the government of Guatemala an outcast among nations because it can provide no evidence that it tries to stop such moments from happening: the coming of the "unknown men," the Squadron of Death, the killers of professors, students, labor organizers, politicians and professionals, peasants and priests considered threats to the political and economic status quo here. Now, they were after a nun.

Such incidents, which add up to a conspicuous "lack of respect for human rights," according to one diplomat here, have made it politically impossible for most Western countries to help the Guatemalan government suppress a growing leftist insurgency.

But to say this country has "human rights problems" does not begin to convey the extent to which terror has become routine here. Amnesty International, the London-based human rights organization, has described "a government program of political murder" directed by Guatemalan President Romeo Lucas Garcia and maintains that death squads are most often composed of soldiers or police. In the largely intimidated local press, however, if the killers are not clearly guerrillas they are called simply desconocidos, unknown men.

Early Jan. 6, Sister Victoria de la Roca lay dying of cancer in her windowless, 6-by-12-foot room. But the rumors had spread in town that she was a subversive, and in the murky world of Guatemalan political violence, that was probably enough to bring the unknown men into action.

People who know her well believe that the drugs she was taking for the cancer, or maybe just the desperation caused by the disease, had unbalanced her. She started talking again about the glories of the Nicaraguan revolution she had lived through. She taught adolescents in her youth group words like "proletariat" and gave them pamphlets that one irate father threw back at her as communist subversion.

These are very suspect things in Guatemala, especially here in the conservative Oriente, Guatemala's wild, eastern area near the borders of El Salvador and Honduras.

Early this month, dynamite was found hidden in a car near the convent. Random violence has become a fact of life here, and frequently no group claims responsibility for it. But after the nun was abducted, the military said she had planned to use the dynamite to blow up the Esquipulas town hall.

Most incriminating of all in Sister Victoria's trial-by-rumor was what she did on the morning of Jan. 5.

With peculiar urgency Sister Victoria had gone to a devout widow here and told her that five seminarians desperately needed a ride to Quezaltepeque.

As the widow got ready to pile everyone into her car she performed a customary ritual. She said the rosary for safe travel. The young men did not know the rosary, which seemed odd for seminarians. People thought maybe the seminarians were really guerrillas.

That night the desconocidos came, and Sister Victoria has not reappeared. Her friends here believe that she probably never will.

In Guatemala City, some human rights watchers were a little encouraged that first week in January, even though it had started badly.

A Belgian priest and a Guatemalan priest were kidnaped and their sacristan was killed by unknown men in Nueva Concepcion, southwest of the capital. Then there had been Sister Victoria's disappearance.

But after vehement protests from Belgian Ambassador Teo Lansloot as well as church officials in Guatemala and the United States, the priests suddenly turned up safe Jan. 7. Something similar happened just before Thanksgiving, when two American nuns along with a Guatemalan priest and seminarian mysteriously disappeared for a few days and then reappeared.

This kind of reappearance almost never happens in Guatemala. The fact that the American nuns and the Belgian reappeared so quickly and without public "confessions" denouncing leftist subversion was seen by some as a marked improvement, at least in Guatemalan terms.

"There may be signs that they Guatemalan officials are starting to see some light. This ought to be encouraged," said a senior Western diplomat. "It may be in many instances better to work quietly than to raise hell, to support those trying to change the system from within, which is not something you do overnight . . . . 'Human rights'--if you say the words here they go crazy and you can't talk."

But other foreign officials concerned with human rights were less sanguine about improvement.

These officials point out that those released are foreigners around whom world public opinion can easily be mobilized, while peasants keep dying in large numbers. They cited another instance of massive torture and murder Jan. 9 with the killing of at least 38 people from the town of San Francisco el Tablon in the northwest region of San Marcos. There were at least 20 killings of 50 people or more reported in the Guatemalan press in 1981.

Many diplomats question Amnesty International's contention that virtually all right-wing killing is directed by the government, saying that after so many years of violence it is often impossible to tell who is killing whom and why. They also note indications that the guerrilla terrorism has increased.

But one foreign official concerned with human-rights problems said that whoever is responsible for the mass killing, in which bodies typically are dumped miles away from the home village, is "somebody who has got the trucks, vehicles to carry them, has the organization and is able to move around in broad daylight" over the heavily policed Guatemalan highways. "Desconocidos," the official said as he smiled.

"You will note," said one increasingly cynical diplomat, "that the chief of staff of the armed forces has said Sister Victoria was deeply involved in subversive activities."

Gen. Benedicto Lucas Garcia told local reporters after the nun's disappearance that 130 sticks of dynamite found in Esquipulas disguised as tamales probably were destined for her. He added that the Army had proof she was in communication with guerrilla commanders.

"I hope she appears safe and sound and makes a formal statement," he said. "As soon as this woman appears, the government will present charges against her to find out who is responsible in this case. I insist: We have credible proof that we're talking about a subversive."

In a later interview, Lucas said, "Many priests and monks are implicated in subversive actions. We've seen it, and we know it positively, and we've established what they are doing. And I recommended that these clergy, so they wouldn't be doing damage to humble Indians whom they oblige to join in the subversion, be pulled out of the national territory . . . ."

Sister Victoria was said to have screamed as the men beat in the door of her room, breaking the doorjamb, and dragged her toward the back of the convent.

Other men ripped the robe worn by Sister Esperanza and forced her and Sister Beatrice, the only other nuns in the convent, to lie on the floor. Meanwhile, others ransacked the convent, spreading gasoline. A fire was set and within seconds much of the convent was engulfed in flame. Finally the men fled.

According to several friends of the nuns who related this part of the story, at one point the apparent leader of the group, wearing a hat and a scarf over his face, very deliberately identified his group to one of the nuns as a unit of the leftist Guerrilla Army for the Poor and himself as "Commandante Arnoldo."

When military investigators came after a few days, according to this account, taking advantage of the opportunity to search the convent for arms, ammunition and subversive literature, one asked the nuns repeatedly, "Did the men identify themselves?" And when the sister was slow answering, he said, "Are you sure one of them didn't say he was a commandante or something?"

If Sister Victoria had ties to the guerrillas she was not alone. Some priests in Guatemala have been completely radicalized by the teachings of "liberation theology" and what they encounter here. At least one has openly joined the guerrillas. At least nine others have been slain during the past two years, including an American, the Rev. Stanley Rother.

Last January Rother wrote to a friend about the increasing carnage he saw in the little village of Santiago de Atitlan. Eleven members of the community had been kidnaped since the Army set up camp nearby, and all were presumed dead, he said.

Rother knew the violence could cut both ways, but he found it increasingly difficult to remain neutral.

"I just got word," he wrote, "that the Army has vacated the camp they had outside town . . . . If it is true that they moved out, then the government informers will now be scared and looking for a place to hide. It could be that some guerrillas will come in now and take care of the leaders, at least of the informers."

The next day guerrillas ambushed a government convoy. Rother wrote that "in retaliation the Army picked up 17 townspeople who were not involved in anything. Their bodies were found in different parts of the country. They, these bodies, were badly tortured, e.g. skin peeled off their faces, etc."

Rother was slain by unknown men in July.