The Supreme Court narrowed the privilege against self-incrimination yesterday by ruling that a defendant's failure to admit what he did can sometimes be used against him in court.

The 7-to-2 decision brought vigorous denunciations from Justices Thurgood Marshall and William Brennan, who called it just another example of the Burger court's "disparagement" of "individual freedoms" in the criminal justice field.

Dennis Jenkins, the defendant in yesterday's case, had taken the stand voluntarily during his Michigan murder trial to argue that he knifed a man to death in self-defense during a struggle. In front of the jury, prosecutors challenged his story, saying that if it ws really self-defense, Jenkins would have turned himself in to police, in effect confessing to the knifing.

Jenkins appealed to the Supreme Court, contending that the prosecutor breached his Fifth Amendment privilege against self-incrimination, which has traditionally prohibited the use of a defendant's prior silence as an adverse inference at trial.

But Justice Lewis Powell, writing for the majority, ruled that Jenkins "cast aside his cloak of silence" by taking the witness stand at his trial, giving up Fifth Amendment protections otherwise guaranteed.

Under these circumstances, Powell said, prosecutors could use Jenkins' silence to impeach his credibility, if not as prejudical evidence of the crime. s

Marshall and Brennan said the "extraordinarily broad" decision "strikes a blow at two of the foundation stones of our constitutional system: the privilege against self-incrimination and the right to present a defense."

The defendent's silence, they said, is unlikely to be relevant to his credibility since there are numerous reasons -- other than guilt -- why he might not have surrendered to police. In addition, a ruling that a defendant "casts aside" protections by testifying at his trial "impermissbly burdens the decision to exercise the constitutional right to testify in one's own defense."

Brennan and Marshall, once part of the Warren court's "liberal" majority in the field of defendants' rights, are now part of a shrinking minority in that area.

Their dissent -- in which they also criticized other Fifth Amendment decisions of the Burger court -- underscored that change. "I disagree not only with the court's holding in this case" Marshall wrote for himself and Brennan, "but as well with its emerging conception of the individual's duty to assist the state in obtaining convictions including his own -- a conception which, I believe, is fundamentally at odds with our constitutional system.

". . . There is on doubt an important social interest in enabling police and prosecutors to obtain convictions. But the court does not serve the nation well by subordinating to that interest the safeguards that the Constitution guarantees to the criminal defendant."

In other action yesterday:

The court extended the protections afforded Vietnam veterans against losses of seniority rights in the workplace during the period of their military service.

Thomas E. Coffy, who brought yesterday's case, came back from the war only to be laid off with other workers at Republic Steel. But he received lower unemployment benefits under a steelworkers union contract because of time away from the workplace. The court ruled that the Vietnam Era Veternans' Readjustment Assistance Act of 1974, which prevents most seniority losses, also applied to the suplemental unemployment benefits in the steel industry. The ruling was unanimous.

A divided court ruled that Indians in the state of Washington have to impose that state's sales tax when they sell cigarettes to nonresidents of the reservations. The Indians, who were able to sell at a discount by imposing only a tribal tax but no state tax, had argued that federal reservations need not follow the state law in such cases.

The court ruled, however, that while reservation residents may be exempt from the state tax, nonresidents had to pay it.