Attorney General Benjamin R. Civiletti vigorously defended his role in the Billy Carter investigation yesterday and denied any impropriety in suggesting to President Carter in June that his brother could avoid prosecution by registering as a foreign agent.

Civiletti said his brief chat June 17 with the president had no effect or impact on the Justice Department's investigation, and said he "had absolutely nothing to do with the Billy Carter matter from June 17 until after" a civil suit forcing Billy Carter's registration as a foreign agent was filed the next month.

At a daylong hearing before a special Senate investigating subcommittee, Civiletti acknowledged, however, that he did make "a serious mistake" in publicly denying on July 24 that he had had any communications with the White House about the conduct of the investigation.

In retrospect, Civiletti added later, he would not have talked to the president if he had it to do over again -- not because it was wrong, but because it "wasn't worth all the hubbub about it."

"I don't think anything I said was ill-advised or improper," the attorney general said under questioning by Sen. Max Baucus (D-Mont.). "But in hindsight, I wouldn't do it again. It wasn't worth it."

The five Democrats and four Republicans on the panel seemed more disturbed, however, by Civiletti's decision in April to keep two highly sensitive intelligence reports about Billy Carter's dealings with Libyans from Justice Department officials in charge of the investigation.

Civiletti said he did not disclose the contents of the top-secret reports, one of which indicated that the president's brother was about to receive some money from the Libyans, because he was afraid of compromising the source of the reports and because he did not want to "abort the transaction" through premature disclosure.

Sen. Strom Thurmond (R.-S.C.) maintained that Civiletti at least could have alerted department investigators to be on the lookout for a transfer of money to Billy. Thurmond suggested that the attorney general was, in effect, "forcing them to look for a needle in the haystack" without even telling them "what haystack" to look in.

Civiletti denied it, insisting that lawyers in charge of an investigation under the Foreign Agents' Registration Act would automatically be seeking a "quid pro quo" as evidence of a foreign power's control over a suspected agent. The attorney general said he was confident that any payments to Billy Carter would "kick up dust" enough for government investigators to find.

"Bills would be paid and deposits made," Civiletti said. "There would be [news] reports that Billy was plush again. We would pick that up."

Subcommittee Chairman Birch Bayh (D-Ind.) pointed out that it did not turn out that way. Civiletti finally turned over copies of two intelligence reports to the head of his criminal division, Assistant Attorney General Philip B. Heymann, on June 4, but only after lawyers assigned to the case learned of the payments from other highly secret intelligence sources.

Confronted a few days later by Justice Department attorneys Joel S. Lisker and Mark Richard, the president's brother admitted getting $220,000 from the Libyans in recent months.

"The good Lord or somebody up there was smiling at us," Bayh observed. "It wasn't the dust we turned up. It was another informant . . . an intelligence source that was no more usable than the first. What broke the case was [Billy] Carter's own admission."

The other intelligence report that Civiletti withheld from his subordinates for six weeks dealt with Billy Carter's attempt to win an increased allocation of Libyan crude oil for the Charter Oil Co. of Jacksonville, Fla., which stood ready to pay him a multi-million-dollar commission if the deal went through.

In contrast to the secrecy with which Civiletti handled the report, White House national security adviser Zbignew Brzezinski, who learned of it a month earlier, phoned Billy Carter about its contents before the attorney general ever learned of it.

It became clear at yesterday's hearing, in turn, that Central Intelligence Agency Director Stansfield Turner served as a sort of special messenger in alerting Brzezinski about the matter.

According to Civiletti, the Libyan oil report did not mention either the Charter Oil Co. or Billy Carter by name. Civiletti said he was orally informed of those points when he first saw the document on April 10.

Civiletti said he also was told by the intelligence official who briefed him that this report, "without the identifications, has been supplied to others." In addition, the attorney general said he was told that of Charter Oil "had been supplied to one requestor."

That could only have been Turner.In a White House report delivered to the subcommittee last month, Brzenzinski stated that it was the CIA director who in March "drew my attention to a brief intelligence report which bore on Billy Carter's commercial dealings with an oil company and Libyan efforts to exploit them."

Brzezinski is now under investigation by the Justice Department's Office of Professional Responsibility for having called Billy Carter that same day in March to talk to him about the contents of the top-secret document.

Brezezinski has said he told the president's brother that "I have recently seen some information which seems to suggest that you are engaged in an oil deal and that you are seeking an increased allocation from Libya for a U.S. oil company." The White House aide said he warned Billy Carter that "this could be exploited politically and it could create considerable embarrassment for this country and for the president personally."

Justice Department officials told the subcommittee on Thursday that they regarded this exchange as "a potential violation of the espionage law" when they first heard about it June 11 while interviewing Billy Carter.

In a statement yesterday about the status of the Brzezinski inquiry, Michael E. Shahenn Jr., head of the Office of Professional Responsibility, said it was part of an effort to determine whether any criminal violations occurred, including "whether anyone in the White House leaked national security information to any unauthorized individuals."

"The scope of the investigation includes Dr. Brzezinski," Shaheen said. "We have not, however, at this time developed any evidence indicating that Dr. Brzezinski has committed any criminal violations."

CIA Director Turner is expected to be invited to an executive session of the subcommittee Tuesday. Brzezinski will be called to testify later this month, staff lawyers said.

The Justice Department's Lisker, who heads the Foreign Agents Registration Unit, told the senators Thursday night that he recently received confidiential "information" that Libya's first payment to Billy Carter stemmed directly from the meeting he arranged last Nov. 27 between Brzezinski and the chief Libyan diplomat here, Ali Houderi.

The White House had asked Billy Carter to set up the meeting in an effort to gain Libya's support for release of the American hostages in Iran. The president himself met with Houderi on Dec. 6. His brother banked a $20,000 check from the Libyans later that month.

"This [the $20,000] was an expression of gratitude and so was the $200,000" from the Libyans in April, Lisker asserted. He declined to elaborate on the source of his information, but said knowledgeable individuals "have indicated that great store was placed by the Libyans on the fact that Billy Carter was able to accomplish" the Nov. 27 and Dec. 6 meetings. Lisker said he believes that the $220,000, which Billy Carter has described as a loan, was actually "compensation amounting to a gift."

In an earlier deposition for the subcommittee, Lisker also voiced suspicions that a 10-day waiting period in the investigation had been decreed by Civiletti last June 11 so that the attorney general might have time to talk to the president before any formal actions were taken.

Yesterday Civiletti called that "kind of a crazy speculation." Denying any devious motives, he said he suggested the hiatus only after Lisker told him on June 11 that he was going to make another pass at getting Billy Carter to register, and that he needed time to develop more evidence in any event.

Aside from the time spent deciding what to do about the intelligence reports, Civiletti estimated that he devoted "less than an hour and probably closer to half an hour of my time" on the Billy Carter case since the investigation of his Libyan dealings started in January 1979.

The attorney general asserted that he decided to speak to the president on June 17 because he thought Billy might have complained about his Justice Department treatment and he wanted the president to understand that "this was one which he should not discuss."

Civiletti said he also told the president that his brother was "a damn fool" for not having registered long ago, but that "if he told the truth and registered under the act, then it was my understanding that the general practice in the department was not to prosecute."

The attorney general contended that this did not in any way amount to "a 'deal' or a 'commitment" of any kind and any suggestion to the contrary are unfair and baseless."

Several members of the subcommittee suggested that Civiletti should have told the president much more, much sooner. Sen. Charles McC. Mathias (R-Md.) said he felt "this whole sorry mess might have been prevented" if the president had been urged early on to set his brother straight. Mathias said he saw no virtue in trying to insulate the White House from such predicaments.

At Watergate hearings, in the same Senate Caucus Room seven years ago, one of the chief complaints was that there wasn't enough distance between the White House and the Justice Department.

Sen. Dennis DeConcini (D-Ariz.) yesterday raised one of the biggest mysteries of the entire investigation: the alleged report of an informant named John Feeney to the U.S. attorney's office in the Southern District of New York on Feb 3, 1979, that Billy Carter was to be offered a 50-cent-a-barrel commission from Charter Oil for an increased supply of Libyan crude. Feeney said he got the information from the fugitive financier Robert Vesco. This was some three months before Billy Carter himself, by all accounts, had any idea of making such an arrangement with Charter Oil or any other company.

Civiletti said he had never been informed of the Feeney report and knew of no one at headquarters here who had been told of it by the prosecutors in New York. He assured DeConcini he would try to find out whether anyone in the criminal division here had been notified.