The Supreme Court, rejecting efforts by the Reagan administration to limit affirmative action programs, ruled for the first time yesterday that judges may order strict racial promotion quotas to overcome "long-term, open and pervasive discrimination."

The 5-to-4 ruling, with Justice William J. Brennan Jr. writing the main opinion, upheld a judge's temporary order in 1983 that required Alabama to promote one black state trooper for every white promoted. The order followed a finding that, 12 years after a judge ordered the state police desegregated, only four of 200 officers were black and none of the blacks was above the rank of corporal.

The decision, the third in the past two years to uphold racial quotas, underscored the Reagan administration's failure after six years to make significant headway in winning Supreme Court support for its civil rights arguments.

The administration had called the quota "excessive" and "profoundly illegal," maintaining that any promotion quotas must be "narrowly tailored" to reflect the percentage of blacks applying for state police jobs. In Alabama's case, that would be about one black for every three whites.

But Brennan, joined by three justices, disagreed, sanctioning the use of the "speed-up" promotion plan. "The one-for-one requirement," Brennan said, was imposed after the judge "properly determined that strong measures were required in light of the {state's} long and shameful record of delay and resistance." Brennan emphasized that the strict quota was used only once and suspended as soon as the state implemented a nondiscriminatory hiring plan.

The administration also had argued that the one-for-one quota "unnecessarily and severely burdens innocent" white troopers. Brennan replied that "because the one-for-one requirement is so limited in scope and duration, it only postpones the promotions of qualified whites."

"Denial of a future employment opportunity is not as intrusive as loss of an existing job," Brennan said, adding that the quota in this case did not "disproportionately harm the interests, or unnecessarily trammel the rights, of innocent individuals."

Justices Thurgood Marshall, Harry A. Blackmun and Lewis F. Powell Jr., joined Brennan's opinion. Justice John Paul Stevens agreed that the quota should be upheld but disagreed with the majority's reasons. Justice Sandra Day O'Connor, Chief Justice William H. Rehnquist and Justices Antonin Scalia and Byron R. White dissented.

O'Connor said there was "no evidence in the record that such an extreme quota was necessary . . . . "

The court generally has approved affirmative action at the hiring level, whether voluntary or court-ordered. At the same time it has virtually ruled out any plan that might result in layoffs of white workers, saying that would unduly harm innocent people.

Last term, the court in one case approved a consent agreement calling for numerical hiring and promotional goals and, in another case, said a judge could order a union to hire a certain number of minorities to meet a specific hiring quota.

Yesterday's ruling in U.S. v. Paradise was the first time the justices have said judges could force employers to use a strict numerical quota in promotions.

The ruling resolves longstanding questions over the general power of judges to order affirmative action once an employer has been found guilty of discrimination.

But other substantial questions are pending before the justices in a case from California, which asks whether a public employer may give preference to minorities and women in promotions without evidence of past discrimination.

Assistant Attorney General William Bradford Reynolds, who has led the administration's effort to curb certain affirmative action plans, said yesterday's decision essentially paralleled one last term involving sheet metal workers in the New York area. Reynolds said that "in the face of proven, persistent discrimination," the high court has carved out a "narrow exception" to a rule against racial preferences where it will "tolerate the use of racial preferences with respect to promotion and hiring."

Barry Goldstein, an attorney with the NAACP Legal Defense and Educational Fund Inc., said the court's use of a "balancing approach bodes well" for civil rights groups in future cases. The court, Goldstein said, rejected the administration's attempt to put "rigid limits" on affirmative action.

Powell, often a pivotal vote in affirmative action cases, joined Brennan's opinion but wrote a separate concurrence explaining that the quota was proper in this case because it was short in duration and the "effect of the order on innocent white workers is likely to be relatively diffuse."

Justice Stevens agreed that the quota should be sustained but disagreed with the court's analysis. Stevens said that, once an employer has been found guilty of discrimination, a judge has "broad and flexible authority" and his "discretion is {not} constricted by" a need to find a "narrowly tailored" solution. The judge's remedy must be upheld unless is "exceeds the bounds of reasonableness."

O'Connor, joined in dissent by Rehnquist and Scalia, argued that the Alabama judge "unquestionably had the authority to fashion a remedy designed to end the {state's} egregious history of discrimination." But O'Connor said the judge was obliged to find a solution "that was narrowly tailored to accomplish this purpose," and she said there was "no evidence in the record that such an extreme quota was necessary to eradicate the effects of the {state's} delay," or to compel the state to develop a nondiscriminatory hiring plan.

O'Connor said the judge should have considered other alternatives to spur action by the state, alternatives that "would have a lesser effect on the rights of {white} troopers."

The district court "did not even consider the available alternatives to a one-for-one promotion quota," such as imposing fines on the state, O'Connor said. White, the fourth dissenter, said he believed the Alabama judge exceeded his power in ordering the quota.

In another decision yesterday, the court ruled 8 to 1 that Florida violated the First Amendment's guarantee of religious freedom when it denied unemployment benefits to a Seventh-day Adventist who was fired from her job after she refused to work on Saturday.

The case, Hobbie v. Unemployment Appeals Commission of Florida, began in 1984 when Paula Hobbie, who had worked in a jewelry store for 2 1/2 years, told her employer that she was to be baptized into the Seventh-day Adventist Church -- which celebrates the sabbath on Saturday -- and that, for religious reasons, she would no longer work from sundown on Friday to sundown on Saturday.

Her boss fired her and she filed a claim for unemployment compensation. Florida officials said she was not eligible for benefits because her refusal to work certain shifts amounted to misconduct connected with her work.

The court, in an opinion by Brennan, reaffirmed its rulings in similar cases in 1963 and 1981. Brennan, quoting a prior court ruling, said the First Amendment does not allow a state to "force an employe 'to choose between following the precepts of her religion and forfeiting benefits.' "

Powell and Stevens concurred in the ruling, while Rehnquist dissented.