An appeals court has ruled that there is currently insufficient evidence to bring five national guardsmen to trial for the 1980 murder of four American churchwomen.

The highly technical decision does not dismiss the charges against the guardsmen, and they will remain under arrest as new evidence is gathered, according to Salvadoran court officials. But the move does represent another setback in the politically sensitive prosecutions at the time when the Reagan administration is seeking major increases in aid for the government here.

Relatives of the victims and opponents of Washington's support for this regime, as well as some State Department officials, have accused the Salvadorans of first trying to cover up the case and now stalling its prosecution.

Both the brother of one of the murdered nuns and the attorney for the families, Michael Posner of the Lawyers' Committee for International Human Rights, are scheduled to testify before Congress Thursday. In a telephone interview, Posner called the latest move here "disturbing, frustrating and a step backwards at a time when the administration insists that there is progress."

The accused guardsmen were first detained two years ago for the crime committed on the night of Dec. 2, 1980. U.S. aid to El Salvador previously has been made at least partially contingent on progress in the prosecution of this and other cases of Americans allegedly murdered by Salvadoran government forces.

A spokesman for the U.S. Embassy here said this afternoon, "We are reserving judgment until we have had a chance to read and analyze the decision."

In Washington, State Department spokesman John Hughes said the embassy's preliminary information was that the judge was "returning the case to an earlier stage" of the judicial process to clarify evidence and "our information is that the five remain in custody and that the case will remain on course."

The terms and procedures of Salvadoran law differ from those in the United States. Under the system here, the accused is presumed guilty from the point at which his case is ordered to trial.

The primary judge in the case ordered it to trial in November, but defense lawyers took the judgment to an appeals court. Its ruling, dated March 11 but not made public until today, revokes the judgment of the lower court that there is sufficient evidence to prove the five accused guardsmen guilty in the murder of nuns Ita Ford, Maura Clark and Dorothy Kazel and lay worker Jean Donovan.

After a review of the voluminous evidence presented in the case, much of it collected by or under the supervision of FBI agents, the court asked for some specific new information without which it says the case cannot be ordered to trial.

Among these new requisites is more information about the burned microbus in which the victims were driven to their deaths. The appeals court wants to know who the owner was and what that person has to say about the crime.

The wording of the current ruling by Magistrate Armando Lazaro Cruz and Luis Alonso Melara leaves open the possibility that even when these questions are answered, the defense appeal could be upheld and the case dismissed.

Posner, who visited El Salvador in January along with William Ford, the brother of Ita, cited the current setback as a demonstration that "there is no good safe effort on the part of the prosecution to get the case in order." Posner estimated that the proceedings could be held up for several more months.