An FBI special agent, who left his family to go undercover and infiltrate the mob for an unprecedented 6 1/2 years, told a federal jury today that his disguise worked so well that the Mafia set him up in a Florida gambling and loan shark operation and ultimately asked him to hunt down and kill the rival leader of the group and "leave him right in the street."

Special agent Joseph D. Pistone, who operated under the name "Donnie Brasco" and who is currently under round-the-clock guard, testified in a racketeering and murder trial in Manhattan federal court here that he infiltrated "the Bonanno family," posing as a jewel thief, in 1976.

Gradually, in his work operating a "bottle club," or unlicensed bar, in Florida, he gained the confidence of the alleged Mafia family sufficiently to be promised membership. Last May, apparently convinced that he could be trusted, he testified he was called to New York City by a Bonanno capo, Dominick Napolitano.

Napolitano, Pistone testified, told him there had been trouble among "the Bonanno family." He said three rivals had been killed, but one--Bruno Indelicato--had gotten away.

"I took care of those three guys--they're gone," Pistone testified Napolitano told him. "He said Bruno had gotten away . . . . He said he had a cocaine habit and he had some connections in Miami. He said, 'I want you to go down to Miami and look for him and find him and hit him.' He said, 'When he's coked up he's crazy, but he's not a tough guy with his hands.' "

Under questioning from prosecutor Barbara Sue Jones, Pistone, a soft-spoken man in a somewhat louder sports jacket, set the record straight on the obvious.

"When you were told to hit him, what did you take his meaning to be?" asked Jones.

"Kill him," Pistone said.

The case on trial here is considered by the FBI to be its most successful infiltration so far of an alleged crime family. Eleven men are under indictment. Five are on trial here, three charged, in addition to racketeering, with conspiracy to murder. Napolitano was not among them. He is missing and in some quarters presumed dead.

The investigation cost the FBI between $1 1/2 million and $2 million, according to an agent who took the stand last week. The defendants, thus far, are bearing up well. "You a member of the Mafia, Boots?" a television reporter asked defendant Antonio (Boots) Tomasulo on the evening news. "Never heard of 'em," Tomasulo cheerfully replied.

In opening statements last week prosecutor Louis J. Freeh said that the five defendants "made their living through robbery, hijacking, narcotics trafficking and gambling . . .and that when a power struggle erupted in this family, it was not settled by vote or negotiation, it was settled quickly and violently by murder."

The five-member defense team countered, however, that the government lacked proof. They also insisted that prosecutors were simply trying to justify an expensive investigation that had produced "no evidence of significance."

There is, in this trial richly sprinkled with street language, with its talk of "wise guys" (mobsters), "made guys" (members) and "hits," an element of farce. The group the FBI targeted, at such expense, has been, according to testimony, the sponsor of not a few failed enterprises.

An attempt to rob the apartment of the sister of the shah of Iran failed when one unfortunate wise guy--actually a career criminal now turned informant--shot himself in the hand. ("It seemed like a piece of cake to us," witness Raymond Wean testified last week, adding that after the shooting there had been a swift decision "to abort the score.")

The man identified during the trial as the immediate boss of FBI informant Pistone, Benjamin (Lefty) Ruggiero, who sits dourly in court, could not, according to testimony from Pistone, afford the $600-a-month rent for the group's Brooklyn social club, and occasionally put the touch on Pistone for a few hundred bucks.