Sidestepping any mention of Billy Carter by name, the Senate agreed yesterday to create a special nine-member subcommittee to investigate has activities with the Libyan government.

The chairman of the new panel, Sen. Birch Bayh (D-Ind.), said he hopes to begin hearings -- as Senate Republicans have urged -- before the opening of the Democratic National Convention in New York Aug. 11.

Speaking at a news conference after the Senate floor action, Bayh said the subcommittee would "pursue the truth wherever it may lead and let the chips fall where they may."

It is submit either a final or interim report by Oct. 4, one month before the presidential election.

The White House pledged its full cooperation, saying that staffers will testify in person if the subcommittee deems it necessary, and leaving open the possibility that President Carter would testify if needed.

Senate Democrats went through a number of contortions to keep the special panel from having the status of the select committee that investigated the Watergate scandals. White House press secretary Jody Powell expressed annoyance Wednesday with congressional Republicans who "compare this with Watergate."

Senate Majority Leader Robert C. Byrd (D-W.Va.) acknowledged, however, that "this is a matter that will not go away."

The president's brother was forced to register with the Justice Department as a foreign agent last week in a statement disclosing that he had received $200,000 from the Libyans this year. His limited disclosures touched off a torrent of news stories bringing additional details to light.

The resolution setting up the investigation, adopted in the Senate by unanimous consent, formally establishes the panel as a subcommittee of the Senate Judiciary Committee. It will have seven members from that committee and two from the Senate Foreign Relations Committee. Five of them will be Democrats and four Republicans.

The vice chairman of the subcommittee will be Sen. Strom Thurmond (R-S.C.), who designated Sens. Bob Dole (R-Kan.) and Charles McC. Mathias (R-Md.) as the GOP members from Senate Judiciary. Bayh told reporters he had yet to make his three selections from that same committee, but he was said to be sounding out Sens. Patrick J. Leahy (D-Vt.), John C. Culver (D-Iowa) and Max Baucus (D-Mont.).

Byrd and Senate Minority Leader Howard H. Baker Jr. (R-Tenn.) will appoint the other two members from the ranks of the Foreign Relations Committee, and were said to be considering Sen. Claiborne Pell (D-R.I.) and Jacob K. Javits (R-N.Y.).

The subcommittee was assigned, without further elaboration, to conduct "an investigation of activities relating to individuals representing the interests of foreign governments." Carter's name was not mentioned until midway through the news conference, when reporters asked why it was missing.

"There no necessity for it," Byrd said. "It [the investigation] might not be confined to Billy Carter."

Asked if there was any chance that the president's brother might not be the subject of the investigation. Bayh responded: "Not unless you don't read the newspaper clippings."

The subcommittee will decide the extent and scope of the investigation. It may meet today after a session of the full Judiciary Committee, where the dickering over the inquiry began Wednesday morning.

At that meeting, Dole proposed that the investigation be conducted by the Judiciary Committee, without further ado, but the Democrats resisted, only to find out later that that was where Byrd wanted to keep it.

Republicans seized on the indecision to press for an independent, Watergate-style select committee, with members picked from the Senate at large, and then settled for the compromise adopted yesterday.

Baker said at the news conference that he was satisfied that "the major elements of a select committee arrangement have been protected." He pointed out that Thurmond as well as Bayh will have the authority to sign subpoenas "at the direction of the Subcommittee."

Baker acknowledged that a straight 5-to-4 partyline vote on the subcommittee could block any subpoena, but added: "My hunch is that the temptation will be to err on the side of more subpoenas rather than too few."

Bayh said he hoped most witnesses would choose to testify voluntarily, without requiring subpoenas.

"We want to resist the temptation to demagogue, to witch-hunt, to destroy people's characters," he said. "But we will pursue the truth."

The subcommittee staff is expected to be a hybrid, composed of aides from the office of Senate legal counsel Michael Davidson (a Byrd appointee), deputy legal counsel Robert Kelley (a Baker appointee), the Judiciary Committee staff and perhaps some outside counsel and investigators.

Judiciary Committee staffers appointed by Chairman Edward M. Kennedy (D-Mass.), who disqualified himself from any role because of his presidential candidacy, are not expected to participate in the investigation.

Byrd emphasized to reporters that he wanted the investigation to be "expeditiously done." He said he felt this could best be done by an existing committee that was "already geared up for investigations." He said he was confident that the subcommittee would do a creditable job.

Other Democrats took comfort in the fact that compromise had resulted in Mathias and Javits looming as likelier prospects for membership on the subcommittee than the more conservative Orrin G. Hatch (R-Utah) and Jesse A. Helms (R-N.C.), who had been mentioned earlier.

"Maybe personalities, as much as anything, had something to do with it," an aide to one leading Democrat said of the negotiations.

Bayh said he would like to start the inquiry next week, if possible, but said he did not want to act so hastily "that we shoot ourselves in the foot." In response to a question, he said, however, that "we should be prepared to start the hearings before the convention."