The U.S. Court of Appeals here yesterday ordered the Nuclear Regulatory Commission to keep the undamaged reactor at Three Mile Island shut down until it studies the effect a restart would have on the mental health of people in the area.

The 2-to-1 decision is a major setback for the financially strapped Metropolitan Edison Co., which owns the Pennsylvania power plant, and for the nuclear industry, which sees the case as precedent-setting. The citizens' group that brought the suit, People Against Nuclear Energy (PANE) of Middletown, Pa., was exuberant.

"We can't wait to see the ruling and really believe it's true," said PANE board member James B. Hurst. "It means it's time to stop talking about psychological stress around here and to start dealing with it."

The ruling overturned an NRC verdict early last year, which was reaffirmed this fall, that psychological effects did not have to be considered in its pending decision on whether to allow the undamaged Unit One reactor to start up. NRC must now conduct an "environmental assessment" of restart effects on "the psychological health of neighboring residents and on the well-being of surrounding communities."

The NRC must then determine whether the National Environmental Policy Act requires a more detailed environmental impact statement, which would include public hearings on the effects of a restart, the court said. The NRC may not decide on reopening the plant until that is done, it added.

In a strong dissent, Judge Malcolm R. Wilkey said the court order set up psychological stress as "an 'impact' which has never before been considered as covered by NEPA." He said it was "yet another example of a court inventing new procedural requirements for an administrative agency in a manner which has enormous substantive consequences."

While the assessment continues, he said, the TMI neighborhood will get no service from the plant, and "it will be a colder winter than predicted."

Unit One was ready to begin operations the day the adjacent Unit Two overheated, on March 28, 1979, spreading radiation over the area, and has been closed ever since for design changes.

But Met Ed said it feared bankruptcy if it is not allowed to restart Unit One soon and apply the revenues toward cleaning up Unit Two. Issues of structural strength, managerial competence and operator test cheating have been pending, but the NRC had been expected to allow the restart within the next few weeks. Yesterday's decision could delay that several more months.

PANE argued that psychological factors should be included in the Atomic Energy Act's requirement that public "health and safety" must be considered in authorizing nuclear power plant operation.