Billy Carter, the president's brother, testified yesterday that he is not "a buffoon, a boob or a wacko." Nor, he said, has he committed any crime by borrowing money or trying to obtain oil from some friends who happen to be Libyan.

As Billy Carter told his story, he was "just an ordinary citizen from a small southern community" victimized by his brother's sudden rise to fame, the constant scrutiny of the press and over-zealous government investigations of his private affairs.

The marble-columned Senate Caucus Room, once the scene of the Watergate hearings, was flooded with television lights and packed wall-to-wall with more than a hundred reporters as Carter sat in the witness chair, dressed in a three-piece suit and flanked by his Washington lawyers.

The former gas station owner, part-time farmer and peanut warehouse operator displayed none of his usual earthy manner. He smoked nervously during the six hours of questioning, responding briefly and respectfully to questions from Senate Judiciary subcommittee investigating his Libyan affairs.

Carter said he still believes he did nit need to register as a foreign agent for Libya because "I never offered or did anything for the benefit of Libya or the government of Libya in relation to American government policy or actions."

By forcing him to register, Carter suggested that the Justice Department regarded him as "a hot potato requiring a stricter form of law-enforcement attention than that afforded an ordinary citizen not related to the president of the United States."

The most dramatic moment of the hearing came when Philip Tone, the subcommittee's chief counsel, emerged from a closed-door session to announce that two of Billy Carter's friends and associates who had helped him obtain a loan from Libya were under Justice Department investigation for marijuana and cocaine smuggling.

However, Tone emphasized that the allegations against the two men, George Belluomini and Ronald C. Sprague of Bakersfield, Calif., included no evidence that Carter was involved.

Sen. Birch Bayh (D-ind.), subcommittee chairman, later called the issue irrelevant to the proceedings.

Carter told the subcommittee that the Libyans did not lend him money because they thought he had any influence over U.S. government policy, a policy somewhat hostile to Libya because of the regime's terrorist activities.

He said he told the Libyans that, "If the people and government of Libya wanted to treat me as a friend, that was fine with me. If the people and government of Libya wanted to deal with me as a man of potential influence, they had chosen the wrong person, and I so informed them."

Carter said the Libyans offered to help him financially only because his income from public appearances dried up after he made a trip to Libya with a group of Georgians in September 1978 and, a few months later, hosted a Libyan trade delegation in Georgia.

"I truly believe that the Libyan people with whom I dealt felt personally responsible for the fact that I had lost my means of financial livelihood," he said. "They were astonished at the outpouring of negative public reaction to their visit to Georgia. They had copies of these negative articles from the American press."

Pressed by several senators about whether he regretted his actions, Billy Carter expressed no remorse. "I don't know if I would do it over again. Before I do anything again with Libya I'd talk real long and hard with my lawyers," he said.

While critics accuse Billy Carter of trying to exploit his relationship with his brother in order to make money, the portrait that emerged from his testimony yesterday was that of a man trying desperately to flee from his order brother's shadow. "The president is not his brother's keeper," he said. "I love my brother. I love my country. And I am my own man."

Henry (Randy) Coleman, Billy Carter's close friend, testified Tuesday that Carter did not tell the president he was going to Libya before his first trip because he thought he might be stopped. Nonetheless, Billy Carter said of his Libyan dealings yesterday, "I never did think it would affect the president or the country."

Asked if he had considered the fact that the Libyan government was responsible for the murder of dissidents, the sacking of the U.S. Embassy in Tripoli, the assassination of Israeli Olympic athletes and other terrorist activities, Carter responded that he had seen such news reports, but that he did not always believe newspaper atrticles, and that "I was treated extra nice by the citizens of Libya" on his trips there.

He reminded the committee that Libya is the United States' third largest supplier of imported oil.

Carter denied that he was under the control of Libya, as several senators alleged, or that the repayment of the $220,000 loan he received was dependent upon the Libyans' granting him an allocation of 100,000 barrels of oil a day, from which he would gain a lucratice commission in a brokerage arrangement with the Charter Oil Co.

He said he was planning to repay the loan partly by selling a piece of property he owned in southern Georgia, recently appraised at $250,000.

In discussing the Justice Department's efforts this spring to make him register as a foreign agent, Carter denied that he had been tipped off that the government had found out about his Libyan loan through intelligence reports.

He said the only reason he called the department in June to check on the status of the foreign agent investigation was because he'd read a newspaper article quoting the attorney general as saying that it was taking a long time.

At the time, he said, he was considering doing a television show in Nashville, and "I felt it would be irresponsible to go forward . . . while the results of a Department of Justice investigation were pending."

When the subcommittee informed him of the smuggling operations against Belluomini, a California farmer and a friend, and Sprague, a financial consultant who had helped him negotiable with the Libyans, Carter seemed genuinely surprised. He told the committee it was "a complete shock."

Nonetheless, Sen. Bob Dole (R Kan.), the sharpest partisan on the subcommittee, issued a news release saying that the allegations "open a whole new dimension in this increasing complex case. . . . At best what we see developing here is a classic case of manipulation -- and an appalling lack of judgement on Billy Carter's part."

Carter said he had talked about Libya with the president. "He has never been there, and we had family talks just as you would tell a member of your family about a trip you may have taken to a foreign land. I never asked my brother to do anything for the benefit of Libya. I also know that he would not do it even if I had been foolish enough to try."

He added, "I remember quite well when Jimmy was governor of Georgia and the state repaved the streets of Plains, with one exception -- the small portion of street in front of my house was not repaved. Neither Jimmy Carter nor I believe that the government can or should be used for the financial benefit of members of the president's family.