The Justice Department announced yesterday that it has closed two investigations into Attorney General William French Smith's personal finances and that no special prosecutor will be appointed to investigate his acceptance of a $50,000 severance fee from a California firm.

But in a second matter, the department's Office of Professional Responsibility issued an unusual criticism of an attorney general. It said Smith had violated the department's conflict-of-interest rules through an investment in a controversial tax shelter that would have given him $4 in tax write-offs for each $1 of his first-year investment.

Soon after questions about Smith's personal finances became public in May, he returned the $50,000 and said he would not take tax deductions greater than his actual cash investments. But the investigations continued in an attempt to determine whether there had been any violations of the law or conflict-of-interest rules.

Smith's spokesman yesterday issued a brief statement saying, "The attorney general is pleased with the unanimous conclusion that these matters have now been closed."

As recently as last week Smith vigorously defended his tax shelter investment and severance fee, saying he gave up financial benefits only to head off Sen. Edward M. Kennedy's (D-Mass.) efforts to turn the matters into a political issue. Kennedy had criticized the tax shelter as "welfare for the rich."

Smith has called the tax shelter "proper" throughout the controversy. But continuing publicity about his personal financial matters have deeply disturbed him, acccording to department officials. In addition, the incidents have led some senior White House officials to question his political judgment.

Despite its criticism, the department's Office of Professional Responsibility, the internal ethics investigators for Justice, said no further action was required on the tax shelter--Yale-Quay Energy Partners--because the attorney general had limited his deductions.

Department rules prohibit any official or employe from making investments "which are reasonably likely to create any conflict in the proper discharge of his official duties."

Smith initially invested $16,500 to get a $66,000 tax deduction eight months ago. He did so despite a warning that the Internal Revenue Service had, in 1980, through Revenue Ruling 80-70, "objected to the very method employed by Yale-Quay to calculate tax benefits," according to yesterday's memo.

The memo noted that the attorney general's duties include supervising the Tax Division, which represents the IRS in federal court. This created a conflict between the private investment and the IRS position, and the memo said, "By electing to go forward with the investment despite the ruling, the attorney general breached the standards of conduct provision . . . ."

The four-page memo, dated Monday, was signed by Michael E. Shaheen Jr., the head of the Office of Professional Responsibility.

The memo continues, "As the nation's chief law enforcement officer, the attorney general has himself acknowledged that he does not stand in the shoes of the average investor. . . .

"The existence of Revenue Ruling 80-70, the validity of which the IRS continues to assert, dictates our finding that the attorney general should not have invested in Yale-Quay.

"The attorney general subsequently took voluntary steps sufficient in our view to cure the substance of this regulatory violation. . . . Accordingly, no further action need be taken regarding these investments. . . ."

Because the investigations involved Smith, and because his top two deputies, Edward C. Schmults and Rudolph W. Giuliani, withdrew from taking part in the investigations, yesterday's announcement was made by Solicitor General Rex E. Lee, the department's fourth senior official.

Lee called the conflict-of-interest finding against Smith a "technical violation," and said it wasn't clear to him that the regulation applied to the facts of the case.

He added, "There is no need to resolve that narrow legal issue, because in any event I agree with the Office of Professional Responsibility that no further action need be taken, and the case should be closed."

Lee said that an FBI inquiry found that the $50,000 severance fee Smith accepted from the Earle M. Jorgensen Co. in January, 1981, was intended as compensation for his past services to the company, not to supplement his government salary.

It is against the law for a government official to accept compensation for his official duties from any outside source. Lee said that the receipt of the payment and the events surrounding it did not trigger the special proscecutor provisions of the Ethics in Government Act because there was no evidence that the money was given in connection with his government job.

An accompanying report by D. Lowell Jensen, head of the department's Criminal Division, said that 13 of company's 14 directors "stated unequivocally" that the payment was for past services "and was not motivated by his future government employment."

One director told the FBI that Jorgensen said in a private conversation that Smith's "situation was different from other resigning directors because Mr. Smith would be making a financial sacrifice in accepting public service.

"However, that director also emphasized that the payment was for 'loyal and dedicated service' to the company."

The report said the board members believed Smith's contribution to the company was valuable and they noted that he was the first outside director who didn't have a business relationship with the firm.

It quoted Jorgensen, a member of President Reagan's so-called Kitchen Cabinet, as saying Smith was a "bargain" because of his six years of service as a board member and member of the audit committee.