The court-marital of Cpl. Jerry D. Rousseau, the Marine accused of raping and murdering an American school teacher in Morocco, opened yesterday in a small, cinder-block courtroom at the Quantico Marine Base with two of the victim's sisters looking on.

Rousseau, who was a guard assigned to the U.S. Embassy in Rabat at the time, is charged with raping and stabbing to death 39-year-old Dorothy Layton in her apartment in the Casbah only two weeks before she was to have rejoined her husband at a Missouri resort owned by the couple.

Rousseau, a boyish-faced 21-year-old, sat pale and solemn through the first day of trial, clasping a small black cross. Behind him sat a row of reporters and Opal Gestring and Rena McClain, sisters of the woman Rousseau is accused of killing.

Rousseau's two Marine lawyers immediately challenged the constitutionality of the death penalty, a legal possibility for the defendant, and the right of the military court to hear the case.

The defense attorneys argued that Rousseau fraudulently entered the Marines four years earlier by concealing his prior use of marijuana and that he therefore did not come under military authority.

The judge, Col. Robert E. Switzer, acknowledged outside the courtroom that if Rousseau can prove the recruiter who enlisted him was aware of his prior drug use and helped him to conceal that fact, as Rousseau contends, the court would be forced to dismiss the case against him.

Switzer also told a reporter that he did not believe a death penalty was likely to be carried out by the service.

Rena McClain, one of Mrs. Layton's sisters, said yesterday that she had wanted Rousseau tried in Morocco because she feared he would escape prosecution through a legal technicality or be released after a short prison term.

Jurisdiction over the case of Cpl. Rousseau has been the subject of dispute between Moroccan authorities and the U.S. military. If the military tribunal is forced to dismiss the case, then Rousseau probably cannot be tried by either the Moroccan government or U.S. civilian courts, Switzer said.

Before the trial began, McClain, who has a heart condition and said she came from California against her doctor's orders, spoke of a suitcase of letters she has amassed over the past 15 years from her sister.

She quoted passages from her sister's letters that she had committed to memory. "I decided to sleep under the stars tonight. I knew I could make it," she said, quoting one such letter.

"I thought she was indestructible," McClain said of her sister, her eyes filling with tears. She said she had received five letters from her sister after she learned of her sister's death.

The trial is expected to last several weeks. Because of a shortage of airplane flights from North Africa and the inability of the court to subpoena witnesses out of the country, the case presents logistical problems for both defense and prosecution.