The Penthouse magazine libel trial demonstrates, if nothing else, that crime does pay--up to $250 an hour.

Penthouse is paying that much to a self-described mob "hitman" to testify as an expert witness against claims that it libeled the owners of a plush California resort by suggesting they were involved in organized crime. But the highly paid Penthouse defense team also includes former agents for the FBI and the Internal Revenue Service, and this alliance has raised troubling questions in the secretive halls of the FBI.

It is not unusual for both sides in a legal dispute to hire expert witnesses, but attorneys for the resort and other sources allege that Penthouse bought more than just expertise from some of these erstwhile federal employes. It is alleged that when ex-agents retired they took with them classified investigative documents bearing on the libel case.

The trial, now dragging into its 12th week in Compton, Calif., Superior Court, stems from an article that appeared in the March, 1975, issue of Penthouse. The story alleged that organized crime figures founded, financed and frequented a plush resort in Carlsbad, Calif., called Rancho La Costa. The owners of La Costa promptly responded with a $490 million libel suit.

Penthouse, seeking to prove the truth of the article, has hired numerous self-described organized-crime figures as expert witnesses and investigators. One expert, self-described mob "hitman" Aladena (Jimmy the Weasel) Fratianno, is on loan to Penthouse from the federal protective custody program, where he is a government witness against fellow mobsters. Fratianno gets $250 an hour from Penthouse while he prepares to testify; his lawyer gets $125 an hour, according to court papers.

The allegation that "confidential information and documents have been illegally taken from the FBI's files" is contained in a May 15, 1981, letter to Associate Attorney General Rudolph W. Giuliani from Washington attorney Judah Best, who was retained as special counsel for La Costa.

As outlined in Best's letter, the controversy over documents dates to November, 1980, when the California court granted a motion by Penthouse attorneys to take testimony from FBI officials in Washington concerning confidential documents.

The Penthouse attorneys requested some 54 documents from the agency. The FBI asked the magazine to get an order from District of Columbia Superior Court releasing the documents.

When attorneys for La Costa belatedly learned of the California judge's order, they went to D.C. Superior Court to oppose release of the documents, claiming the information requested was far in excess of what the judge approved. The motion was denied and in May of last year the documents were released to Penthouse.

Shortly after, Best wrote Giuliani asking the Justice Department to seek an order vacating the earlier Superior Court decision. He argued that Penthouse had the documents before it asked the FBI for them, and was going through the civil discovery process "to legitimize or cover up the theft of this information from the FBI.

"The nature of documents . . . and the specificity with which they are described indicate that the security and confidentiality of the FBI's records have been breached," Best wrote.

Government agencies--particularly investigative agencies like the FBI and the IRS--are extremely careful not to release documents that are classified or that would violate the privacy act. Moreover, the names of federal investigators are invariably expunged from released documents.

But a review of the requests for documents by Penthouse, on file in D.C. Superior Court, shows that the magazine's attorneys had unusually specific information in advance, including the names of FBI agents.

The request included, for example:

* "Identification record of Morris B. Dalitz [a La Costa founder] under FBI No. 4 124 252."

* "Report of Special Agent W. Albert Stewart Jr., dated Apr. 19, 1958, at Salt Lake City, entitled 'Morris B. Dalitz, anti-racketeering.' "

* "Report of Special Agent Nicholas J. Lore dated May 14, 1975, at San Diego, entitled 'Rancho La Costa Inc., et al.' "

* "San Diego airtel [referring to an expedited internal communication] to director dated Nov. 5, 1965, entitled 'Rancho La Costa Inc., et al. , anti-racketeering.' "

Norman Roy Grutman, a New York criminal attorney who is arguing Penthouse's case in court, angrily denies any suggestion of wrongdoing. "If you are suggesting some alleged impropriety, I find the suggestion offensive," he said by telephone from Los Angeles. "We didn't have the files in advance."

Other sources familiar with the case said that, during the years of pre-trial work by Penthouse, attorneys for the magazine told them they were getting cooperation from FBI agents, some of whom went on the Penthouse payroll after retiring.

The Penthouse article, "La Costa: the Hundred Million Dollar Resort with Criminal Clientele," was written by two freelance investigative reporters, Jeff Gerth, now with The New York Times, and Lowell Bergman, with the American Broadcasting Co. Last year, the co-authors settled with the plaintiffs and are no longer defendants in the suit.

The suit was filed by La Costa, four of its subsidiaries and two of its founders, claiming that the article falsely linked them to organized crime. The two suing founders are Merv Adelson and Irwin Molasky, who, after establishing the huge resort complex on the Pacific Ocean north of San Diego, also started Lorimar Productions, which turned out television shows such as "The Waltons," "Dallas" and "The Blue Knight."

But Penthouse's attorneys have zeroed on another founder, who is not a plaintiff in the current proceeding, 82-year-old Morris B. Dalitz, a Las Vegas casino proprietor. Dalitz has acknowledged in testimony before the California jury that he was a bootlegger and liquor smuggler during Prohibition, had operated illegal gambling houses in the Midwest and knew or was friendly with a long list of organized crime figures.

The ex-FBI agents working for Penthouse have been drawn from bureau offices in California, Nevada and the Midwest and are under the direction of John R. Barron. Before he retired from the FBI in 1978 after 27 years, Barron had supervised agents working with the Los Angeles federal organized crime strike force.

In a telephone interview, Barron was asked if he or other ex-agents on the Penthouse defense team had brought any documents from the FBI. "Not to my knowledge," he said. But, he said, "What's in my head is in my head."

"I know who Moe Dalitz is. I can tell them attorneys for Penthouse what records to subpoena. But I wouldn't have remembered specific dates."

Penthouse is subpoenaing agents whose names appear on the contested documents, causing some concern among law enforcement officials that court questioning will get into classified matters. But Grutman, the Penthouse lawyer, says, "At no instance are we going to blow anybody's cover."

Grutman, who has a reputation for aggressive questioning and courtroom showmanship, got into trouble recently for his trial behavior. In a separate case involving Penthouse, U.S. District Court Judge Thomas P. Griesa in the southern district of Manhattan sanctioned Grutman for "willful misrepresentation to the court." The opinion was upheld last October by the Second Circuit Court of Appeals, which found that Grutman had made "material misrepresentations" and had been "grossly negligent."

As for the La Costa case, Grutman says "bureacratic inertia" has stalled the war against organized crime, a wrong that he says the civil suit against Penthouse will help correct. "I've spent seven years on this case," he says, "and I thank God for the opportunity to shake the sleepy head of the law."