James Neal, a former Watergate special prosecutor, yesterday turned down the job of investigating Billy Carter, telling Senate special subcommittee Chairman Birch Bayh (D-Ind.) that he could not spare the time from his private law practice in Nashville.

Neal's refusal was a major setback for the fledging investigation, which has been plagued by an atmosphere of partisan politics and the shadow of the coming election, as well as by dwindling enthusiasm among senators on the subcomittee.

Yesterday, most of the nine senators had left town or were about to return to their home states for a 10-day recess, thus throwing into doubt whether a chief counsel could be found in the next week. Bayh has said he wants to wrap up investigative hearings within three weeks, a timetable which now seems increasingly improbable.

A tense dispute arose this week between the subcommittee and the White House, which was insisting on having White House lawyers present during interviews of President carter's staff members.

However, Bayh said he talked yesterday afternoon with White House counsel Lloyd Cutler and "he understood it is not wise to let something like that give the appearance of trying to inhibit witnesses on their testimony."

Bayh added, "Cutler said he would work it out and find a way for volunteer attorneys to sit in" with those who must be interviewed.

Sen. Richard G. Lugar (R-Ind.), in an interview yesterday, criticized "the tedious and glacial place" of the investigation, adding that the longer it takes to find a chief counsel and get the effort organized, "the more we intrude on the election and the more partisan it could become."

Lugar said, "The majority of subcommittee members are serious about this investigation, but we don't want to make it a lifetime career. We need to have the staff counsel working in the next 10 days . . . The majority party can either be timely [about conducting the inquiry] or string it out through the fall."

Bayh plans to spend the recess in Indiana, but he said "we're going to bust our tail trying to find" a chief counsel in the next week or so. He interviewed one candidate by telephone yesterday, identified by one source as former U.S. attorney Robert Fiske of New York.

Bayh said the publicity about Neal being offered the job was unfortunate, because "whoever is next is number two or three choice, and that lessens the chance of getting someone."

Meanwhile, Bayh may have to contend with a morale problem among the more than a dozen attorneys who have already been assigned to the investigation from senators' personal and committee staffs. "No one has a clear concept of what to do," said one top aide to a Democratic senator on the subcommittee.

"It's a turkey in terms of being a real investigation. The president's news conference took the wind out of our sails. The remaining issues are questionable. You're getting into real technical judgement calls. We're not seeing any laws broken -- maybe a little fudging here and there. It's not all that clear cut," the aide said.

The aide, who asked to remain anonymous, said part of the problem stems from the fact that the inquiry is not set up in a separate body like the Watergate committee, but is a subcommittee of the Judiciary Committee, and thus has a staff of young Senate aides who, for the most part, have no investigative experience.

"They don't have enough back-ground to ask the right questions," the aide said. "They wouldn't know what's meaningful unless it bit them in the ear." Another problem, the aide said, is that most of the subcommittee investigators do not have access to critical classified data, which is being carefully guarded in the Senate Intelligence Committee.

Bayh, however, strongly defended the pace of the investigation, saying that he had signed between a dozen and three dozen subpoenas for witnesses, and that interviewing will proceed this weekend.

As for the qualifications of the staff attorneys, he said, "We don't need Sam Slade private eye. Most lawyers know how to ask questions . . . They aren't lightweights."

Coordination between the subcommittee and the Intelligence Committee is no problem, Bayh said. While intelligence sources must be protected, he said, "Most of the information I've seen [in the classified documents] has been in the newspapers. It comes from several different sources. It can be handled so the committee members can know to facts."

Bayh also pointed to the hiring as chief investigator of retired FBI agent John McDermott, once the No. 3 man in the bureau. McDermott said in a telephone interview that he is satisfied with the staff, which "is working hard to do the job under adverse circumstances and within a very difficult time frame."