The Supreme Court rejected a Reagan administration appeal yesterday that was intended to free the government from carefully monitoring Social Security payments to persons unable to manage their own affairs.

The justices cited lack of jurisdiction in refusing to hear arguments that keeping track of such payments will overload the federal bureaucracy needlessly.

In other action, the court:Declined to revive an Indiana school district's plan for protecting black teachers from layoffs. The court left intact a ruling that South Bend school officials violated white teachers' rights when they dismissed them rather than blacks with less seniority. Killed a suit by the widow of a man who died after receiving a heart transplant from a donor with a different blood type. The justices barred a Missouri woman from suing in her home state a Texas hospital that provided the organ for transplant. Allowed a transsexual Indiana prison inmate, who says he is being denied adequate medical care and protection, to sue state officials. The court let stand a ruling ordering further study in the case.

In the Social Security case, Jeanne A. Jordan of the Oklahoma City area sued the government over $2,456 in benefits she said were spent unlawfully by her sister.

Jordan suffers from schizophrenia, and the money went to her sister in her capacity as the recipient's representative.

In a second test case, Barbara Leeds of Provincetown, Mass., who also suffers from a mental impairment, said her former roommate misspent nearly $9,000 of her federal benefits.

U.S. District Court Judge Lee R. West ruled in 1985 and again this year that federal officials must require an accounting from representatives of such beneficiaries.

The administration said his rulings, if applied nationwide, would affect more than 3 million people who receive the benefits and are represented by others because they are unable to manage their own affairs.

The Justice Department said the lower court rulings could "generate potentially burdensome and needless paperwork" for the Social Security Administration and for representatives managing the benefits.

The Health and Human Services Department, which administers the Social Security program, proposed a system of random checks of 10 percent of the recipients.

West rejected that idea, ruling that due process demands that all representatives of those receiving benefits make an annual accounting. The 10th U.S. Circuit Court of Appeals threw out the administration's appeal of the ruling last January.