Citing its attorney-client relationship with the White House, the Justice Department has refused to turn over to a House subcommittee its legal opinions on a controversial presidential advisory panel that is staffed and funded by private business.

But the chairman of the subcommittee, Rep. William D. Ford (D-Mich.), isn't giving up. Yesterday he threatened to subpoena Attorney General William French Smith, if necessary, to get the documents on the President's Private Sector Survey on Cost Control.

In a letter to Ford late last month, Assistant Attorney General Robert A. McConnell acknowledged that the department's Office of Legal Counsel had prepared legal opinions for the White House on "various issues" raised by the creation of the advisory panel. But he declined to make those opinions available, saying that they "constitute sensitive, deliberative, attorney-client advice."

Ford, chairman of the House Post Office and Civil Service Committee and its investigations subcommittee, said he found McConnell's response "difficult to understand."

"If we were seeking classified defense data or documents on foreign policy, I could understand the use of the word 'sensitive,' " he said. "But what can possibly be 'sensitive' about legal matters pertaining to a study of management practices in the federal government? . . . Clearly there is something they don't want us to find out."

The exchange is the latest in a series of clashes between Congress and the administration over the private-sector panel, which President Reagan created with an executive order last June 30 to help ferret out waste and inefficiency in government and to advise the president "with respect to improving management and reducing costs." Ford's subcommittee, concerned that the panel was dealing with policy questions rather than government efficiency, conducted two days of hearings in September and commissioned a General Accounting Office investigation of the panel.

Officials with the Cost Control Survey, which is headed by J. Peter Grace, chairman of W.R. Grace & Co. and a friend of the president's, initially refused to give the GAO a list of the more than 1,000 members of its 35 task forces, which are assigned to look into the activities of specific agencies. In an interim report to the subcommittee last month, the GAO suggested that the administration was going out of its way to protect the task force members from federal conflict-of-interest laws. The list, which has since been made public, shows dozens of executives assigned to the agencies that regulate their industries.

"Without question there is at least the appearance of conflicts of interest," Ford said yesterday.

The subcommittee believes the Justice Department documents will help answer what Ford called "disturbing questions" about whether the private-sector panel was "intentionally structured to circumvent the Federal Advisory Committee Act and the conflict-of-interest laws."

Ford yesterday accused the administration of "attempting to stonewall" the committee's investigation. "The administration has gone to great lengths to put extensive and unprecedented security wraps around the activities of the task forces and the management staff," he said. "One would have to be blind not to be suspicious."

Ford has renewed his request to the Justice Department, and department spokesman Art Brill said yesterday that "we're looking it over" in consultation with the White House. Brill said the administration "obviously would like to avoid a confrontation" with the subcommittee, "but we'd like to maintain a deliberative posture."

If Ford carries through his threat to issue a subpoena, the attorney general would join a growing club of Reagan administration officials who have been called on the carpet by Congress for failing to respond to requests for information.

Environmental Protection Agency administrator Anne M. Gorsuch has been subpoenaed to appear before the House Energy and Commerce Committee when Congress returns in late November for failing to turn over files in three hazardous-waste cases. The same committee voted in February to hold Interior Secretary James G. Watt in contempt of Congress for failing to deliver some documents on mineral leasing, prompting White House counsel Fred Fielding to order the documents turned over.