A federal judge, rebuffing government efforts to prosecute a major defense firm, has dismissed criminal charges accusing Litton Industries Inc. of seeking to defraud the Navy of $37 million in overcharges for building three nuclear submarines.

The ruling, by a U.S. District Court judge in Mississippi, marked a dramatic setback for U.S. prosecutors in a controversial investigation that has been under way since the mid-1970s. The allegations centered on cost overruns in construction of three nuclear attack submarines between 1968 and 1971.

Adm. Hyman G. Rickover, shortly before retiring from the Navy a year ago, cited delays in the Litton probe as evidence of the Justice Department's "poor record" in prosecuting shipbuilders. Justice, Rickover contended, "is systematically closing down these investigations--either overtly or by inaction." The admiral complained that prosecutors had "made no effort" to bring Litton to trial in recent years.

Judge Adrian G. Duplantier, a Louisiana jurist temporarily assigned to hear arguments in the Litton proceedings in Biloxi, Miss., issued an order dismissing the criminal indictment last week. He gave no explanation for his ruling, but said he would furnish his reasons later.

Litton's attorneys had argued that the charges should be dismissed partly because of "inexcusable and prejudicial" delays stemming from the government's failure to bring the allegations to trial. The indictment was initially returned by a grand jury in Northern Virginia in April 1977.

Company lawyers contended that their ability to defend Litton against the charges had been prejudiced by the deaths since 1977 of seven potential witnesses, including two who died in the past two years. They also accused government officials of losing several boxes of documents, which they said were needed for preparing their defense.

A spokesman at Litton's Beverly Hills, Calif., headquarters declined to comment yesterday, saying, "We prefer to wait until we see the rest of the information from the judge."

Assistant U.S. Attorney Joseph A. Fisher III, a Northern Virginia prosecutor, and Frank W. Dunham Jr., a former prosecutor called in by the government as a special assistant U.S. attorney to take part in the Litton proceedings, both said yesterday that they plan to urge Justice officials to appeal the dismissal to the U.S. Fifth Circuit Court of Appeals.

In an interview, Dunham expressed agreement with some of Rickover's contentions about shipbuilding investigations. "The case is so complicated. It shows the difficulty the government has in prosecuting something like this," Dunham said.

The new ruling marks the second time the indictment has been dismissed. A federal judge in Alexandria threw out the indictment in May 1977, citing abuses by prosecutors. But the charges were later reinstated by an appeals court in Richmond, and the proceedings were subsequently transferred to Mississippi, where the submarines were built.

The $100 million shipbuilding work was carried out by Ingalls Shipbuilding, a division of Litton's defense-contracting subsidiary, Litton Systems Inc., at the firm's Pascagoula, Miss., shipyards. Officials say the company previously received about $17 million for cost overruns in a tentative agreement with the government.

In its criminal allegations, the government accused Litton of painting a "totally false picture" to secure $37 million in overruns. It charged that the company made a "deliberate effort to retroactively ascribe to the government responsibility for Ingalls' own deficiencies" in construction practices. The company attributed the overruns partly to delays in steel deliveries caused by the Navy. Prosecutors alleged Ingalls' problems were "unrelated" to the late steel deliveries.

Ingalls denied the allegations.