Attorney General Benjamin R. Civiletti admitted yesterday what he and the White House had repeatedly denied earlier -- that he had discussed the Billy Carter case with President Carter.

Civiletti, in a formal statement, said that on June 17 he advised Carter that his brother Billy probably would not be prosecuted if he registered as a Libyan agent.

The Justice Department's internal watchdog, the Office of Professional Responsibility, immediately opened an inquiry into whether Civiletti or the president broke any law or department regulation by discussing the case. The investigation apparently will focus on whether Civiletti's comments could have triggered the president's call July 1 to urge his brother to register, and thereby possibly obstructed an ongoing investigation that could have led to criminal charges.

Civiletti's surprise admission came just a day after he had stressed to reporters that there had been no contact between the White House and the Justice Department about the case. White House press secretary Jody Powell and counsel Lloyd Cutler had said the same thing Tuesday.

The attorney general told reporters yesterday that he did not feel he lied about having a conversation with the president because he did not consider the discussion to be about the substance of the investigation of Billy Carter.

He said he decided to disclose the talk because the president had said he wanted "every nit and twiddle brought out." He also said Carter told him Thursday night that the president wrote a note after the discussion. "If it was significant enough for the president to make a note about it, I figured it was significant enough to tell you about."

On Thursday, the president pledged to cooperate fully with a planned Senate investigation of Billy Carter's dealings with Libya. Such a note likely would be among the relevant materials the Senate would seek in its inquiry.

Civiletti's revelation is sure to provide more ammunition to administration critics who have been trying to compare the handling of the Billy Carter case to Watergate. Civiletti and the White House repeatedly have emphasized the department's independence from the type of White House interference practiced by the Nixon administration during Watergate.

Civiletti's statement yesterday deepened further the political problem the Billy Carter affair is causing the president as he prepares for next month's Democratic National Convention in New York City.

The controversy was triggered last week when Billy Carter, under pressure from Justice Department, registered as an agent of the government of Libya and disclosed that he had received $220,000 in payments from Libya.

He said the payments were installments on what was to have been a $500,000 loan from Libya to him. The Justice Department declined to accept his characterization of these payments as loans.

Disclosure of the Libyan payments to Billy Carter was followed by revelations that last November the White House used Billy Carter to set up a meeting between Libyan diplomat Ali Houderi and national security adviser Zbigniew Brzezinski. This was followed less than two weeks later by a meeting between Houderi and the president.

The Brzezinski-Houderi meeting was described as part of the effort to free the American hostages in Iran. But Billy Carter's role in setting it up has raised questions of whether the White House, deliberately or not, enhanced the president's brother's stature and therefore his bargaining power with Libya.

Billy Carter received the first payment of $20,000 in January, the month after President Carter met with Houderi.

Another major line of inquiry growing out of the affair has centered on whether Billy Carter may have been informed by someone in the administration that the Justice Department had learned of the $220,000 in payments, prompting him to begin negotiations that led to the filing of the registration statement.

At the White House yesterday press secretary Jody Powell spent the day trying to sort out details of the Carter-Civiletti conversation. Late in the day, he gave this explanation:

Sometime on Thursday, while reviewing his records and telephone logs covering this period, the president came across a written item in his files that reminded him of his "brief exchange" with Civiletti on June 17.

That night, Carter contacted White House counsel Lloyd Carter and told him to ask Civiletti if he also recalled the convesation. Cutler did so, leading to yesterday's statement by the attorney general.

Although Civiletti told the president that if his brother agreed to file as a foreign agent of Libya he would not likely be prosecuted by the Justice Department, Powell said that the president did not pass this information to Billy Carter when he called him on July 1 and urged him to register and disclose his financial dealings with the libyans.

Powell said he saw nothing wrong with Civiletti informing the president of this Justice Department policy on prosecutions since it had been publicly reported in news accounts, was known by Cutler and the White House and undoubtedly was also known by Billy Carter's lawyers.

In his statement Civiletti said he raised the issue with the president at the end of a half-hour meeting in which judgeship nominations were discussed. He said he told the president he could not discuss the investigation, but did not explain, if that was the case, why he brought it up.

He said he thought "Billy was foolish for not having registered long ago." The president then asked him what would be likely to happen under the law if his brother registered. "Based on my understanding of department practice, I told him that if a person tells the truth and registers, the previous failure to register has not been prosecutable."

Civiletti told reporters he did not consider his response to the president's question to be legal advice. "I just thought he [Billy] was a damn fool and I told the president that," he said.

The attorney general said he did not consider the conversation an attempt to resolve the case. "I didn't strike any deal and I don't think the president was under any impression that any deal had been struck," he said.

He did add in hindsight, he probably should not have even discussed that much about the case with the president.

In his statement, Civiletti continued to contend he had no "discussions" about the case with the White House. He called the conversation on "informal, brief exchange."

One department attorney who has worked closely with Civiletti said last night: "I know it's kind of hard to sell. But I can see how Ben could believe there's a distinction there. He has very much a lawyer's approach to everything."

Other department officials were shocked at the disclosure and some wondered privately whether Civiletti can survive the storm that seems sure to follow.

An internal Justice Department investigation of the attorney general and the president is not unprecedented. In early 1978, Michael E. Shaheen, Jr., head of the Office of Professional Responsibility, undertook a similar inquiry after allegations that then-attorney general Griffin B. Bell and President Carter may have obstructed justice in firing a Republican prosecutor whose office was investigating Democratic congressmen.

Shaheen's investigation then showed that neither the president nor Bell was aware of the investigations at the time they decided to sack David W. Marston, the U.S. attorney in Philadelphia.

That inquiry took affidavits from Bell but only an unsworn letter from the president, and its procedures were criticized.

Earlier this week, Shaheen sharply criticized both Civiletti and the White House in his office's 1979 annual report for unexplained delays in carrying out the recommended firings of a U.S. marshal from Kentucky.

Shaheen's office, under the direction of Richard Blumenthal, the U.S. attorney for Connecticut, is not finishing a search for department prosecutors and FBI agents who "leaked" details of the undercover Abscam investigation of congressman to the press in early February.

The revelation of the Carter-Civiletti convesation was extraordinarily embarrassing to the White House, whereofficials have issued repeated statements in the last few days that there have been no contacts between them and the Justice Department on the Billy Carter case. Powell, who has been battered with questions throughout the week, appealed to reporters yesterday to understand that he and his colleagues in the White House are working under the same "two conflicting pressures" that often confront journalists in their work.

"We want to get the information out as quickly as possible, but we also want to be as accurate as we possibly can be," he said.

"We are dealing with personal recollections of events that took place months and in some cases years ago. In some cases, we will provide information later than you like and later than it turns out to be for our own best interests. In other cases, we have had to amend or even correct previous statements.

"In your business as well as ours, the pressures of time and accuracy sometimes result in mistakes. In your institutions as well as ours, it seems to me that mistakes ought not automatically to be considered the result of malice or evil intent."

Yesterday, there was another example or slight variations in information when Powell -- to the surprise of a number of reporters -- casually mentioned a second telephone conversation last month between the president and his brother.

Powell said the president called Billy Carter on June 28 and for about three minutes discussed with him the status of his negotiations with the Justice Department.

The White House "white paper" on the controversy mentions the president's call to his brother on July 1 but makes no reference to the June 28 conversation. Powell said yesterday he had publicly discussed the June 28 conversation, but this apparently was only with a handful of reporters following one of his formal briefings on the Billy Carter affair.