Billy Carter ended a brief period of uncharacteristic silence today and chatted with reporters about his financial problems and his involvement with Libya.

For days, the president's embattled younger brother has been either unavailable or unwilling to discuss his troubles.

"My lawyer said don't talk to anybody," he told a reporter Saturday at his early-morning hangout, the dining room of the Best Western Motel here. "The next people I talk to will be in the Senate." A special Senate investigation of his affairs will begin this week.

But today Billy Carter was talking -- and joking -- once again. He was friendly and relaxed with the reporters who ringed his small table, but unwilling to answer a number of questions.

Carter said he failed to pay income taxes for 1978 and 1979 because federal agencies investigating his affairs have the financial records he needs to calculate how much he should pay. He said he filed a return with standard deductions -- but no payment -- in 1978 and filed no return in 1979.

To provide collateral for the money he owes the government, Carter in April signed over a deed to the IRS on the 58 acres, valued at about $196,000, where he lives. "I gave them the deed on the house just to satisfy them," he said.

Carter said he borrowed $220,000 from the Libyan government -- part of a half-million dollar loan -- earlier this year because American banks will not lend him money. The banks have refused to face the expensive and time-consuming investigations that lending him money invites, Carter said. a

"The bank examiners have come into every bank I've done business with in the last five years," he said.

Before receiving the Libyan money, the last money Carter borrowed -- $25,000 in 1979 -- came from a friend, Georgia realtor Donald Carter.

The president's brother said he borrowed money from the Libyans at the "highest interest raate."

"I wish I could renegotiate," he added.

Carter has said he needed to borrow money because he now has no substantial income, large expenses and major debts.

"Billy Carter stays in financial trouble," he said today. "Doesn't everybody else?"

In comments that broke little new ground in the controversy that has swirled about him and his brother's administration since he registered as a Libyan foreign agent earlier this month, Carter:

Acknowledged that establishing a corporation to arrange trade between the United States and Libya had been "knocked around" but called the specific details recalled by an Atlanta realtor "a complete shock to me."

Said he has no idea what he will do in the future. Because of the investigation, "There's no way I can make plans from one day to the next. The only thing I know is I'm waiting for guys with coats and ties on to come to me with subpoenas." He said he has not had any contact with the Senate panel.

Repeated his belief that his registration statement, filed with the Justice Department, represents a "full disclosure" of his activities for the Libyan government. "The whole thing is so damned simple that nobody believes it. The Libyans have never asked me for anything -- nothing whatsoever.

"The whole thing goes back to Watergate," Carter said. "Every reporter in the country has a Watergate complex."