The Supreme Court yesterday came within one vote of agreeing to consider a politically volatile "reverse-discrimination" case but let stand a lower-court opinion that states may undertake purely voluntary affirmative-action programs.

Three justices, one short of the number necessary to take a case, said they wanted to hear an appeal by white New York state prison guards, who claimed that an examination-grading system for promotion discriminated against them because it increased minority candidates' scores.

Justice William H. Rehnquist, joined by Chief Justice Warren E. Burger and Justice Byron R. White, said the federal appeals court in New York "reached questionable conclusions" in permitting the grading system without proof that the state prison system discriminated against minorities in using the test.

In 1979, the Supreme Court upheld voluntary affirmative-action programs by private employers as long as the action was taken "to eliminate conspicuous racial imbalance in traditionally segregated job categories."

But the court, which last June signaled substantial opposition to affirmative-action programs, has yet to rule whether states or state agencies may voluntarily establish such programs.

The three justices said they wanted to rule on that issue. "States should not be allowed to practice racial discrimination anew under the guise of atoning for past discrimination or because of the difficulties with mounting an otherwise legitimate defense to a lawsuit," Rehnquist said.

The "interests of innocent" persons should be protected under affirmative-action programs, and that will not happen "if agencies charged with discrimination may simply cave in to allegations" of discrimination, he said.

Rehnquist noted peculiarities in the case that might have dissuaded other justices from voting to hear it, but said he still thought the case, Bushey v. N.Y. State Civil Service Commission, should have been reviewed.

Without comment, the justices let stand a ruling that upheld Philadelphia's system of transferring teachers to achieve racial balance among city school faculties.

White teachers challenging their transfers said the city's racial quota system, established in 1978 under intense federal government pressure, was unconstitutional racial classification.

The teachers said that racial balance demanded by federal civil-rights officials had been achieved and that the quota system should be abandoned. A federal judge agreed, but the federal appeals court in Philadelphia last July ordered the program continued. The case is Kromnick v. School District of Philadelphia.

In other cases, the justices:

* Turned down Hustler magazine's appeal of a $150,000 jury award to a Texas woman whose nude photograph appeared without her permission. Hustler argued that it was not at fault because it believed that the photograph was submitted by the woman's husband and that the magazine did not know the photograph had been stolen by a neighbor who submitted it and forged the woman's name on a release form.

The case is Hustler Magazine v. Wood.

* Let stand a ruling by the U.S. Court of Appeals here that judges should not summarily dismiss complaints filed by prisoners against the D.C. Board of Parole without requiring at least some response from the board.

A U.S. District Court judge here threw out a complaint by a prisoner who said the board unfairly forced him to wait for his parole longer than other prisoners in similar cases. The case is D.C. Board of Parole v. Brandon.