Louis J. Carter, the administrative judge heading an investigation of the Indian Point nuclear power plant outside New York City, has resigned from the case, charging that the Nuclear Regulatory Commission has tried to silence critics of the plant.

The NRC yesterday said it accepted with regret the resignation of Carter, who was named last September to chair a three-member panel probing critics' charges that the power site poses an unreasonable danger to the 17 million people in the New York City area about 50 miles away.

The critics say that construction defects in the site's two nuclear reactors make them liable to accidents and that it would be impossible to evacuate successfully the number of people who would be endangered by any accident there.

The five-member NRC, controlled by three Reagan administration appointees, had picked Carter "as somebody tough, a lion tamer, who would keep the crazies in line," said Ellen Weiss, attorney for groups trying to shut down Indian Point.

Carter, 58, a former chairman of the Pennsylvania Utility Commission and Temple University law professor, was regarded as firmly pro-nuclear, Weiss indicated.

But, in his letter of resignation Wednesday, Carter said the independence of his atomic safety and licensing board had been "needlessly subordinated to the commission's other goals" by a series of NRC procedural orders.

Staying in the job, he wrote, "would be incompatible with my sense of fairness" since "we do not share a common concern for the processes which regulate the resolution of these matters."

NRC orders, sent July 27 after being approved on a 3-to-2 vote, told Carter's board to review all of the evidence it had heard in its first month and then accept only evidence supported by testimony that the alleged problems were likely to lead to accidents. Such expert testimony is expensive and normally beyond the critics' resources.

The board was also told to focus testimony on emergency preparedness within a 10-mile radius of the plant in Westchester County.

In an interview, Carter said the July 27 orders were "an unfair burden in my mind on the critics " and were "the straw that broke the camel's back." He added, "It was clear that the commission really did not want us to hear the witnesses."

He stressed that his resignation implies no judgment on merits of the critics' case but only "on what they were going to have to do to present their case."

The July 27 orders "would have had the effect of throwing all our contentions out the window," Weiss said.

Carter's letter also expressed concern about another NRC order in July that halted a different board's consideration of quality control methods at the Zimmer nuclear power plant in Ohio and about outside contacts that NRC commissioners had allegedly had with parties to that dispute.

He complained about "the commission's unwillingness to meet with the board in order to resolve administrative differences."

Acting NRC chairman John Ahearne said in his brief acceptance letter that he regretted Carter had felt it necessary to resign and that the NRC hoped to appoint a new chairman promptly "so as to minimize delay in proceeding."

Commissioners Victor C. Gilinsky and James Asselstine, dissenting from the July 27 orders, wrote that they were "unnecessary, unwarranted and inappropriate intervention by the commission" and that they "may be incompatible with the investigatory nature of the proceeding."

"The majority felt that the board had let too many things into the hearing, and they wanted to cut back. They just told 'em to throw it all out and do it over again," Gilinsky said in an interview yesterday.