The Supreme Court yesterday stepped into one of the country's biggest civil rights controversies, agreeing to review an affirmative action order requiring that layoffs of Boston police and firefighters be made according to race rather than seniority.

The order, designed to preserve jobs for blacks and Hispanics hired after the city was judged to have discriminated in municipal employment, nullified Massachusetts civil service "last hired, first fired" laws and resulted in whites with greater seniority bearing the brunt of layoffs during voter mandated budget cuts in 1981.

The lower court action, combined with a similar order applied to public school teachers in Boston, has been targeted by trade union, state and local officials, all of whom face record numbers of layoffs and reductions in force.

It is the fourth important "reverse discrimination" case to be heard by the court in its history.

The justices also agreed yesterday to hear a dozen other controversial cases, including a major nuclear regulatory case involving the Three Mile Island plant, a decision allowing members of the armed forces to bring civil rights damages suits against superior officers, and a ruling declaring unconstitutional the opening prayer of the Nebraska legislature.

The actions yesterday, added to the court's already lengthy agenda for the term, assure an unusually contentious year for the justices at a time of increasing internal divisiveness.

The Boston cases (Boston Firefighters Union vs. Boston Chapter, NAACP, et al, etc.) stem from successful discrimination complaints brought by black and Hispanic applicants against the police and fire departments during the early and mid 1970s.

The federal courts held then that "racial discrimination had led to the virtual exclusion" of blacks and Hispanics. The courts ordered remedial actions that brought about dramatic increases in minority employment: from 2.3 percent in the police department to 11.7 percent, and from .9 percent in the fire department to 14.7 percent.

When the city confronted massive layoffs in 1981 due to "Proposition 2 1/2," a budget-cutting referendum similar to California's Proposition 13, the minorities went back to court for protection from a state civil service seniority law. A federal District Court, upheld by the 1st U.S. Circuit Court of Appeals, complied.

The courts said that layoffs according to seniority would set back minority progress and ordered the city to avoid any reductions in the minority percentages when it cut its work force, even though whites with seniority would have to be sacrificed.

The laid-off workers have since been rehired.

In other actions yesterday:

* The justices agreed to government and nuclear power industry requests for review of a novel decision last May requiring a psychological distress "impact statement" before allowing one of two reactors at Three Mile Island to restart.

The reactor in question (in Metro Edison, et al, vs. People Against Nuclear Energy, etc.) was not damaged in the March, 1979, accident, but its restarting was delayed indefinitely by leaks and corrosion in thousands of steam generator tubes.

The unprecedented 2-to-1 decision, written by U.S. Court of Appeals Judge J. Skelly Wright here, said that "the anxiety, tension and fear" involved in restarting the reactor, particularly the fear of a "recurring catastrophe," must be taken into account by nuclear regulatory officials before permitting start-up, just as they assess the more obvious potential for physical damage to people and the environment.

The Atomic Industrial Forum, a nuclear power industry umbrella organization, told the court that the decision "adversely affects every nuclear power plant licensee in the country."

The U.S. Chamber of Commerce said the decision was a "major expansion" of the National Environmental Policy Act that could interfere with non-nuclear industrial projects as well by creating a "continuing obligation" for impact statements.

* The court said it would review a $10 million job discrimination suit (Chappell vs. Vernon Wallace, et al) brought by five black sailors aboard the USS Decatur in 1978. They accused their superior officers of assigning them the least desirable duties, excluding them from training programs, giving them low performance evaluations and punishing them with unusual severity for minor transactions.

The officers argued that they were immune from such suits, but the 9th U.S. Circuit Court of Appeals rejected the contention, citing a series of Supreme Court rulings allowing civil rights damages suits against high public officials for constitutional violations.

The lower court noted that suits would not be allowed in "unusual circumstances," such as challenges to combat decisions or decisions of military tribunals. Otherwise, it said, the officers enjoy only a limited immunity, so long as they act in "good faith."

The government told the justices that the ruling poses "a significant potential threat to military discipline" and "undermines the hierarchical relationships that are unique and essential to the military."

Soldiers forced to observe atomic bomb tests during the 1950s have recently raised similar questions about suing superior officers.

* The court agreed to review the Nebraska Legislature's practice of paying a Presbyterian minister to open its daily sessions with a prayer. A member of the legislature successfully challenged the prayer reading on the grounds that it violated the Constitution's prohibition on establishment of religion by a state.

The justices indicated they would focus on the fact that a clergyman from only one denomination always presents the prayer. The decision in Marsh vs. Chambers could affect legislative prayers in 20 states.