Senate Democratic and Republican leaders last night promised an investigation of Billy Carter's Libyan activities, but said they are still dickering over how to conduct the inquiry.

The negotiations, focusing on creation of a special committee that might be composed of selected members of several standing Senate panels, broke up inconclusively after a meeting in the offices of Senate Majority Leader Robert C. Byrd (D-W.Va.).

Byrd said "there will be an investigation," but declined to take any questions from reporters.

Senate Minority Leader Howard H. Baker Jr. (R-Tenn.) said he still favors creation of a select Watergate-style committee.

According to Sen. Strom Thurmond (R-S.C.), however, the Republican offered to settle for a special committee composed of seven members of the Senate Judiciary Committee, two from Foreign Relations and two from Governmental Affairs.

Thurmond said the Democrats said that would be too unwieldy. He said they want the inquiry to be conducted entirely by hand-picked members of Judiciary, where Thurmond is the ranking Republican.

Efforts to reach agreement are to be renewed today. Thurmond said the issue may have to be resolved on the Senate floor.

The talks are touched off by demands for an inquiry by Sen. Bob Dole (R-Kan.) yesterday morning at a meeting of the Judiciary Committee.

Members of the Judiciary Committee had agreed informally at their morning meeting on the need for a thorough investigation of the Billy Carter case, but Democratic senators on that panel insisted on checking with the leadership before launching an inquiry of their own.

Taking over the gavel from Chairman Edward M. Kennedy (D-Mass.), who disqualified himself, Sen. Birch Bayh (D-Ind.) appointed a four-member panel to consult with Byrd, Baker and other Senate committee chairmen.

"I think this matter needs to be laid to rest," Bayh said. "The quicker we can lay this to rest, the better."

Alluding to scattered calls for a special prosecutor, Bayh said there would be no need for one "if we do this job right."

He suggested that there might be some difficulties with an inquiry by the Judiciary Committee alone, since that committee's staff is "primarily Kennedy's."

"Mr. Carter's entitled to due process" Bayh said.

Kennedy disqualified himself because he is still a candidate for president. He said he did not feel he should participate "in making decisions of this sort, involving President Carter, his family or his immediate aides." o

Saying that he, too, felt "very uncomfortable" because he is chairman of Republican nominee Ronald Reagan's campaign committee, Sen. Paul D. Laxalt (R-Nev.) also recused himself.

Billy Carter was forced to file last week as foreign agent for the revolutionary Libyan government of Col. Muammar Qaddafi, which according to the Carter administration, has "repeatedly provided support for acts of international terrorism."

Billy Carter disclosed that he had received $220,000 from the Libyans earlier this year. He also reported serving as an intermediary on a proposed deal involving increased allocations of Libyan oil to a Florida company with a huge refinery in the Bahamas.

At its meeting yesterday, the Judiciary Committee also approved a resolution setting the stage for a possible full-blown investigation -- including hearings and subpoenaed witnesses -- "into alleged contacts between . . . officers and employes of the United States" and financier Robert L. Vesco, a fugitive in the Bahamas.

The officials said to have been contacted -- via a group of their friends in Georgia -- were White House aides Hamilton Jordan and Richard Harden. Vesco has claimed that the Georgia group offered to fix his legal problems in the United States after he had transferred to them securities worth at least $10 million.

The resolution assigned responsibility for the Vesco investigation to the subcommittee on improvements of judicial machinery, and was requested by its chairman, Sen. Dennis DeConcini (D-Ariz.). He must now decide whether there is enough substance to the allegations to warrent a full-scale inquiry. Sources say he may try to speak to Vesco personally before making that decision.

Sen. Orrin G. Hatch (R-Utah), a member of DeConcini's subcommittee, said, "There may be problems in the Vesco case that may be problems in the Billy Carter case." He said, "We do have some information -- sketchy at best -- that the two may be intertwined."

Over the July 4 weekend, Vesco met in Nassau with Hatch, columist Jack Anderson, aides to DeConcini and Hatch, and the full committee's top minority staff investigator. In talks over a 2 1/2-day period, however, according to one of those who was present, Vesco never mentioned the name of Billy Carter.

Bayh appointed himself, Dole, Thurmond and Max Baucus (D-Mont.) to the special subcommittee for the Billy Carter controversy.

Dole, who had called for a Judiciary Committee investigation, at first urged his colleagues "to say we're going to do it or not going to do it" and settle the details later. He wants the inquiry to start before next month's Democratic National Convention in New York. But he relented to the Democratic requests for a brief delay.

"I'm not here to push anybody into a buzz saw," Dole said. "That will be coming fast enough. I was chairman of my party during Watergate, but I hasten to add that was my night off."

Democratic senators such as Patrick J. Leahy (Vt.) and Joseph R. Biden Jr. (Del.) agreed on the need for an investigation, but Leahy voiced fears of a rash of them by separate committees, all stumbling across one other. Biden suggested that creation of a new select committee, which a number of Senate Republicans had already endorsed, mijght be the best approach.

In other developments on Capitol Hill:

The Justice Department's assistant attorney general for legislative affairs, Alan A. Parker, informed the Senate to a request from Dole, that the files of its foreign agents' registration unit in the Billy Carter case would be available to the committee in a day or two.

Republican Reps. Dan Quayle (Ind.) and Bud Shuster (Pa.) called for appointment of a special prosecutor. "It is too much to ask us to believe that the Libyan government paid Billy hundreds of thousands of dollars because of his deep understanding of the oil industry gained through his operation of a service station in Plains, Ga.," Quayle told reporters.

House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) seemed less perturbed. "Rare is the family that doesn't have a black sheep," he said.

Sen. Jesse A. Helms (R-N.C.), a member of the Senate Foreign Relations Committee, suggested an investigation by that committee on the grounds that the Billy Carter case raises the question of whether the Libyans "sought to influence the formulation of United States foreign policy in an improper manner."

The chairman of the Senate Governmental Affairs Committee, Sen. Abraham A. Ribicoff (D-Conn.), and its ranking Republican, Sen. Charles H. Percy (Ill.), turned aside proposals for an inquiry by their committee. Ribicoff said he felt the Judiciary Committee would be "the appropriate forum."