A Salvadoran judge, in what he called "a major step" toward the possible conviction of five Salvadoran National Guardsmen accused of murdering three American nuns and a lay worker in 1980, made an official determination today that the imprisoned guardsmen "participated in the crime" of aggravated homicide.

Under the Salvadoran legal system, the ruling signed today puts virtually the entire burden of proof on the defendants to establish their innocence through a process of appeals and finally an open hearing before five jurors, according to the judge, Bernardo Rauda Murcia. His court is in Zacatecoluca, about 45 miles southeast of the capital near the fields where the nuns were raped, murdered and buried in a shallow grave on the night of Dec. 2, 1980.

Rauda Murcia's ruling comes just a day after a charge by families of the victims that the price of convicting the five guardsmen may be a "coverup" of any involvement by officers or government officials superior to the corporal accused of leading his four subordinates into the crime.

"We have been told that an anticipated defense of the five guardsmen will be that they were only acting under orders of their superiors," said a letter reportedly sent to the State Department by the New York-based Lawyers Committee for International Human Rights, which represents the families.

"Lawyers have warned us that the only way to guarantee a conviction of the five guardsmen is to refute their claim that others were involved," the letter continued. "We reject this specious reasoning. The families insist on a thorough investigation of the case no matter where it leads."

In response to questioning, Rauda Murcia said there was no evidence in his files that established orders from superior officers.

U.S. officials have said privately since the suspects were arraigned in February that they had no case for establishing the involvement of superior officers and have conceded that there is little chance the case would or could be pursued further.

Murcia said that he would not expect the open hearing, at which the defendants would appear and oral arguments by the prosecution and defense would be heard, to take place before January.

The next deadline for the Reagan administration's certification of human rights improvements and progress on this and other cases of murdered Americans in El Salvador is Jan. 28.

After prime suspects in the killing of two U.S. labor experts in early 1981 turned out to be friends of assembly leader Roberto D'Aubuisson and were released by another judge last month, U.S. Ambassador Deane Hinton made a vehement speech before this country's businessmen in which he decried "the absence of an effective system of criminal justice."

Hinton warned that U.S. aid to El Salvador could be cut off if what he called the "gorillas" and "Mafia" involved with human rights abuses are not stopped.

Rauda Murcia, wearing corduroy jeans and desert boots as he sat in his rundown chambers on a Zacatecoluca side street, said in an interview with three reporters this morning that the key evidence in the case against those accused of killing the churchwomen is the testimony of Salvador Rivera Franco, another guardsman.

Rivera Franco said he stayed behind with a car while the other members of his squad, all dressed in civilian clothes, took away Roman Catholic Sisters Ita Ford, Maura Clarke and Dorothy Kazel and lay worker Jean Donovan. The women's bodies were found the following day.