A Senate Judiciary subcommittee cited Attorney General William French Smith for contempt of Congress yesterday for failing to produce internal Justice Department records about its investigation, closed without action in 1981, into whether General Dynamics Corp. submitted fraudulent cost-overrun claims to the Navy during the 1970s.

The contempt citation, signed by Republican Sen. Charles E. Grassley of Iowa in the name of the Judiciary subcommittee on administrative practice and procedure, is not legally valid until it is approved by the full Judiciary Committee and the Senate.

The likelihood of the Republican-controlled Senate taking such action when it meets next year appears minimal.

In an apparent attempt to head off action by Grassley, Judiciary Committee Chairman Strom Thurmond (R-S.C.) refused to schedule a hearing yesterday by Grassley's subcommittee, forcing him to issue the contempt finding during a hearing of Wisconsin Sen. William Proxmire's Joint Economic Committee subcommittee on international trade, finance and security economics.

Grassley's subcommittee had voted Oct. 5 to authorize him to take action to enforce its subpoena of the records.

The citation was labeled "out of place," because it was issued in the name of one subcommittee during a hearing of another, and "uncalled for" by Assistant Attorney General Stephen S. Trott, who appeared before Grassley and Proxmire. Trott told the senators that the Justice Department was still examining their request for internal department records. White House legal advisers are considering whether to assert a claim of executive privilege over the material.

The Justice Department does not want to release the material because it has reopened an investigation in the wake of new allegations by a former General Dynamics executive vice president, P. Takis Veliotis. Veliotis has provided the department with documents and secret tape recordings he made of conversations with other General Dynamics officials from 1977 to 1981.

The subpoenaed records relate to the 1981 decision by the Justice Department to terminate a four-year criminal investigation of General Dynamics, the nation's largest defense contractor. The investigation focused on whether General Dynamics fraudulently claimed that the Navy was responsible for more than $800 million in cost overruns on 18 nuclear attack submarines built since 1971.

The company filed cost-overrun claims in 1975 and 1976 for $220 million and $544 million, respectively. It was preparing additional claims when, in June 1978, senior Navy officials settled the dispute to avoid what they said would be years of litigation and a delay in deploying the attack submarine fleet as well as the first Trident ballistic missile submarines.

Under terms of the settlement, General Dynamics has been paid $639 million in additional compensation for the attack submarines, which cost the Navy about $150 million each compared with a cost of about $100 million each from the Navy's other supplier.

During a hearing yesterday, Richard Kaufman, chief counsel of Proxmire's Joint Economic Committee subcommittee, said Justice Department officials have informed the Senate that twice in 1980 and twice in 1981 Justice Department attorneys working on the General Dynamics case recommended that a criminal prosecution be mounted.

Kaufman also said that in November 1981 the FBI submitted a statement to the Justice Department recommending that General Dynamics' submarine division, Electric Boat, and "two individuals" be indicted.

Trott has told the senators that the earlier investigation into General Dynamics was frustrated by an inability to show criminal intent on the part of General Dynamics officials who prepared and submitted the cost-overrun claims to the Navy. He also vigorously defended the professional judgment and motivation of the career Justice Department lawyers who supervised the case and who ultimately recommended that it be closed.

Trott yesterday urged senators to hold off their request for the documents, saying, "We're going to give you book, chapter and verse on General Dynamics when the case is closed."