A unanimous Supreme Court, rejecting appeals by the Reagan administration, yesterday ordered Health and Human Services officials to reopen the cases of 10,000 mentally disabled New York residents to see if any of them were denied benefits by a secret and illegal procedure.

The high court, in a ruling that could cost the federal government up to $50 million, said that HHS, between 1978 and 1983, secretly changed eligibility criteria in order to illegally restrict benefits.

In addition, the court rejected arguments by the Justice Department that would have blocked the federal courts from hearing appeals from disabled persons challening the denial of benefits.

''We are aware that administrative inconvenience may result from our decision today,'' Justice Lewis F. Powell Jr. wrote for the court. ''But the secretary (of Health and Human Services) had the capability and the duty to prevent the illegal policy found to exist by the district court.''

Powell said those denied benefits ''were denied the fair and neutral procedure required by the statute and regulations, and they are now entitled to pursue that procedure.'' This was not, he said, a ''hard'' case.

In other rulings yesterday, the court:

Acting in a major challenge to the ability of cities to regulate the cable television industry, ruled unanimously that cable operators have certain First Amendment rights but declined to decide what those rights are.

The court, in a unanimous opinion by Justice William H. Rehnquist, said further hearings were needed to determine whether the city of Los Angeles acted properly when it denied a franchise to a cable-TV operator.

Ruled 9 to 0 that an organization of Indiana dentists violated federal antitrust laws by refusing to turn over patients' x-rays to insurance companies trying to contain costs.

The opinion by Justice Byron R. White upheld a Federal Trade Commission order telling the dentists their refusal to turn over the x-rays was an illegal restraint of trade.

The New York disability case began in 1983 when New York officials and a group of mentally disabled filed a class action challenging an internal, unpublished HHs policy that assumed a person could work, and therefore was not eligible for benefits, if his mental impairment did not satisfy certain criteria. Those suing said the policy ignored individual needs.

Congress ultimately changed the policy, but the government argued that it was too late for those cut off to regain their benefits through the courts because they had not filed appeals within a 60-day time limit required by the law.

City and state officials in New York, joined by the American Civil Liberties Union, the American Bar Association and 24 states, urged the court to allow judicial review to those who were denied benefits. The organizations said that the courts had the authority to review the appeals even though many of the disabled had failed to meet the 60-day statue of limitations for filing appeals and had not completed their administrative appeals.

Powell, upholding the appeals court, said federal courts are not always required to defer to administrative agencies ''where the equities'' require judicial intervention.

Since the disabled could not attack a policy that they could not be aware existed, Powell said, ''it would be unfair to penalize (them) for not'' completing their agency appeals. The time limit for appeals was properly extended, he said, because ''the government's secretive conduct prevent (ed) plaintiffs from knowing of a violation of (their) rights.''

Leonard Rubenstein, an attorney for the Mental Health Law Project, which represented the disabled, said that, as a practical matter, the decision was significant more as a rejection of efforts to limit judicial review of broad class-action suits challenging HHS policies.

Rubenstein estimated that Social Security may be forced to pay between $20 million and $50 million to the mentally disabled in New York as a result of yesterday's ruling in Bowen v. City of New York.

The cable television case, Los Angeles v. Preferred Communications, Inc., involved a challenge to the city's use of a competitive bidding process, much like that used in most cities, to award an exclusive cable franchise.

A federal appeals court had struck down the process, saying it violated the First Amendment rights of cable operators that failed to win a franchise. The lower court had compared cable operators' First Amendment rights to those of newspaper publishers.

The court, in a brief opinion that left both sides claiming victory, did not go that far, ceclining to set a precise legal standard for analyzing the constitutional issues.

''We are unwilling,'' Rehnquist said for the court, ''to decide the legal questions posed . . . without a more thoroughly developed record of proceedings in which the parties have an opportunity to prove'' the disputed facts in the case.

''I don't know how you could hope to do any better,'' said Harold Farow, Preferred's lawyer. ''The court said a cable publisher can test government regulations in court . . . .''

But Larrine S. Holbrooke, a lawyer for Los Angeles, said the 7-page opinion, ''while not a slam-dunk win,'' is still a victory given that the appeals court's broad opinion on the First Amendment was not accepted. ''We are pretty confident,'' she said, that the city would be able to justify its regulations at trial.

The court, both sides seemed to agree, left most of the important issues for another day and another case.

In other action yesterday, the justices declined to lift a ''gag order'' on lawyers for former FBI agent Richard Miller forbidding them from speaking to the press. Miller is charged with passing government secrets to the Soviet Union.

The justices, filing the calendar for oral arguments next fall, said they would hear two cases asking the court to make a ruling retroactive in some cases. The ruling last month in Batson v. Kentucky barred prosecutors form using their power to strike jurors for no stated reason in order to keep blacks off juries trying a case with a black defendant. The cases of Griffith v. Kentucky and Brown v. U.S.

The court also agreed to decide whether jurors may be instructed not to consider sympathy or public opinion in determing whether to sentence murderers to the death penalty or to life in prison. The justices, in agreeing to hear California v. Brown, will review a ruling last year by the California Supreme Court that such an instruction was improper.