The Senate Judiciary Committee issued a stinging staff report yesterday that accused the FBI of creating a conspiracy in its investigation of fugitive financier Robert Vesco and his attempts during the Carter administration to secure the export of planes to Libya.

The report said the investigation should never have been started and then was bungled in a confusing tangle of competing informants and targets who did not even know Vesco when it began.

The inquiry eventually produced allegations involving Democratic National Chairman John C. White, White House chief of staff Hamilton Jordan and even President Carter in a purported conspiracy to take a bribe in connection with release of the embargoed Libyan planes.

Then, while appointment of a special prosecutor was being considered, the report said, the FBI refused a request for funds for an informant working for the U.S. attorney's office in the Southern District of New York, partly on grounds that "he was not an FBI informant."

Finally, at the close of the 90-day deadline under the Ethics in Government Act, the report said, the Justice Department "recommended that a special prosecutor not be appointed, relying in part on the purported failure of the undercover investigation during the period when the FBI denied funds to the principal informant in the case."

The 56-page report was written by a special counsel for the committee, John P. Flannery II, and grew out of efforts by Sens. Orrin G. Hatch (R-Utah) and Dennis DeConcini (D-Ariz.) to find flaws in the Justice Department's handling of politically sensitive cases.

The study said the Vesco investigation began when FBI informant James Brewer, who had helped in the exposure of offshore bank scandals, contacted a Texas politician, James Day, and the two somehow got around to talking about arranging a meeting with Vesco even though "neither Brewer nor Day knew Vesco when the subject arose."

It took four months to set up the meeting at which Vesco "set the agenda" and "sought to reverse a State Department ban against the export of planes to Libya," the study said.

Although the FBI refused funds for the Southern District's informant in the face of warnings that this would eliminate his effectiveness, the report said, the bureau a few days later authorized spending $25,000 to offer a bribe to White as part of its Abscam investigation.

The report was also highly critical of a Southern District prosecutor who, it said, told White of evidence that posed "serious inconsistencies" with his grand jury testimony, "information that people in your position are not usually made privy to."

Saying the investigation "very obviously . . .should never have begun," the report contended that, when the probe began, a special prosecutor should have been appointed to find out "independent of the investigation's tortured history, what happened." The allegations involving the White House, the report said, were sufficiently specific.

The FBI had no immediate comment.