Jimmy Flynt is worried. All this violence could hurt business, cut into the sales of his mail-order sex paraphernalia firm, Leasure Time Products.

"The average, square guy doesn't want to know from ali this," says Jimmy Flynt, younger brother of Larry, founder of Hustler magazine. "But the media ties it into a business and Larry Flynt-and that's damaging in the business world."

It has certainly been damaging to the victims.

In the past eight months, Larry Flynt and two men connected with his empire have been struck by violent attacks. And the attempt to make sense of those assaults has only aggravated the controversy surrounding the nation's best-known pornographer since his publishing career began four years ago and since he announced he'd found God after numerous meetings with Ruth Carter Stapleton late last year.

The story of the Flynt family business is a bitter and complicated one that has pitted brother against brother, created rival factions within the various companies spawned by Hustler magazine, and reached such a point that last week Larry Flynt convinced the two most recent victims to undergo lie detector tests.

The latest attempted murder occured last week in Columbus. Jimmy Flynt and William Abrams, his close business, left their warehouse to-friend and associate in the sex-goods gether, walking toward their cars in the early evening darkness.

As Abrams entered his car, a bullet from a high-powered rifle tore through his chest, missing his heart by three inches. He managed to stumble to Jimmy Flynt's Lincoln Continental. During he three-minute drive to a nearby hospital, Abrams, convinced that he would die, asked his friend to "take good care of my family."

But the next day, as a cleaning service worked to remove the blood stains from the blue crushed-velvet interior of Flynt's car, the docoors announced that Agrams would live.

Two months ago, William Rider had been more fortunate. Rider, Larry Flynt's brother-in-law, is employed as regional sales manager for Flynt Publications, with headquarters in Columbus.

On Sept. 14, outside Rider's suburban Columbus home, several sticks of dynamite exploded beneath his Toyota Land Cruiser. No one was injured by the bombing.

Both incidents followed the attempts last March to kill Larry Flynt in Lawrenceville, Ga., where he was on trial for obscenity charges. That shooting left him paralyzed below the waist, and today Flynt lives in Los Angeles, the new headquarters of Hustler's editorial offices.

In the wake of the evidence, theories run wild.

Is organized crime angry at Flynt's intrusion into the skin publishing market? Are religious fanatics furious with the peculiar nature of Flynt's much-publicized religious conversion? Or, as Flynt charged from his hospital bed, is the federal government trying to quash his enthusiastic hunt for a conspiracy int he JFK assassination? or, stranger yet, could the violence have its origins in the Flyn organization itself?

Law enforcement personnel investigating each of the three Flynt-related incidents are baffled. And a million-dollar reward offered by Larry Flynt's wife. Althea, has so far failed to flush out her husband's assailant.

Meanwhile, within the Flynt family, an unpublicized, private split developed between Larry Flynt, his brother and several top business advisers. In the late '60s and early '70s, the two brothers were inseparable, running their go-go bars and beginning their sexually explicit magazine. It was Jimmy Flynt, for example, who traveled to Italy with $35,000 in traveler's checks to negotiate the purchase of grainy pictures of Jacqueline Onassis sunbathing in the nude - pictures that first brought Hustler national notoriety.

But Larry's announcement that he was "born again" last winter threw his staff into confusion and his companies into chaos. He began an exhausting schedule, flying around the country in his private jet to meet with prominent Christians including former Iowa senator Harold Hughes, Charles Colson and Stapleton. He switched to a health food diet and encouraged his employes to follow suit. He became fascinated with conspiracy theories of history. He began spending corporate funds at what his brother thought was an alarming rate to donate to religious and political causes or to buy new properties.

According to Jimmy Flynt, last winter his brother took $2 million out of the company to do such things as buy newspaper ads asking Americans to pray for Hubert Humphrey's recovery from cancer: pay $625,000 in Christmas bonuses to employes: and pay $60,000 a year to hire ex-Realist editor Paul Krassner as Hustler's pusblisher. (Krassner resigned several months ago.)

When Flynt Publications began experiencing some cash flow problems, Jimmy Flynt and several company executives consulted with psychiatrists and then went to court to try to commit their chief executive to a mental institution. Larry Flynt, fasting and praying in the Bananas, learned of the attempt and was furious: he wouldn't return to Columbus until the judge, at Jimmy Flynt's request, tore up the order.

"I never spent a more miserable year in my life," says Jimmy Flynt, who is five years younger than 36-year-old Larry.

Both brothers agree the following the attempted legal action. Jimmy Flynt was excluded from Hustler's inner circle. And it was at that time that the apparent feud began within the Flnt organization.

Some executives, particularly Larry Flynt's wife, Althea, didn't protest their boss' frenetic new schedule of acquisitions (he bought weekly papers in Altanta, LOs Angeles and Plains); others predicted financial disaster for the company employing several hundred people. Jimmy Flynt sided with the second faction and was eased out of day-to-day management along with several other executives who had supported his move to commit Larry Flynt for psychiatric observation.

Jimmy Flynt was skiing in COlorado the day that his brother was shot and he flew to Georgia for a tearful reunion. But it didn't last: in June, the two men separated their business interests.

Larry Flynt kept Hustler, Chic and Ohio magazines as well as his magazine distribution company. Jimmy Flynt took charge of Leasure Time, which he says is the country's largest mail-order dealer of sex and drug-related products. By contract, he receives 10 free pages of advertising each month in Hustler.

But Jimmy Flynt says his brother doesn't want to talk to him.In Larry Flynt's memo to his Columbus office spelling out who was to receive his new California home telephone number.Larry reportedly didn't include Jimmy.

"It's not that I don't want to talk to him," Larry Flynt said last week. "I just want him to stand on his own two feet. Let's face it, I was kind of cool to him after what they tried to do to me last winter . . . but how can anyone be a spiritual person and hold a grudge? I don't want to be doing that. I've been giving him advice all his life - I want him to do it by himself."

Larry Flynt says he can't explain the violence against people in his companies:

"I've tried to think of what it could be. This is only speculation, but maybe Hustler and Leasure Time Products could be cutting into all the underground porn, maybe we're hurting those guys."

He pauses.

"But I know all those guys personally."

William Rider, victim of the unsolved auto bombing, is married to Althea Flynt's older sister, which places him in the Larry Flynt faction. Abrams, now recovering from the gunshot wound to his chest, works for Jimmy Flynt. Rider makes no secret of his dislike for Abrams and Jimmy Flynt says Abrams holds Rider in the same regard.

Despite their mutual animosity, neither man is a suspect in the case, according to Columbus police. But late last week Larry Flynt talked to both Rider and Abrams and convinced them to submit to polygraph examinations.

There are possible explanations for both incidents. Shortly before his car exploded, Rider - a former Columbus narcotics officer - testified at an inquiry into police behavior, alleging that a former fellow officer gave false testimony that led to a murder conviction. That, Rider agrees, generally doesn't win anyone friends.

And Abrams, who once served time in prison for larceny, had a history of arrests during what Jimmy Flynt terms his "tough street days in Dayton." Though Jimmy Flynt says Abrams "hasn't squeezed a grape in years," he may have some lingering enemies.

The Lareenceville shooting seems unconnected to the incidents in Columbus. In fact, the methods were different in all three, suggesting that no methodical effort is being made to kill Flynt family and friends.

Still, the fear of some principals, and the dislike on the part of some employes for one side or another, is real - one reason Larry Flynt wants Rider and Abrams to take polygraph tests.

In his new Bel Air home, Larry Flynt is installing an alarm system. He keeps an attack dog nearby and is beefing up his security as he begins to travel again.

The Riders installed outdoor lighting at their Columbus home: they keep the lights on all night. They plan to move to Los Angeles soon - not solely because of the bombing, although Rider admits that meident made their decision easier.

Jimmy Flynt is moving his and Abram's parking spaces to a more secure location, hiring a private guard and installing more lighting around the warehouse tlat houses his rubber goods, explicit posters and cocaine-sniffing paraphernalia.

"I can't see a pattern," says Larry Flynt. "And that's just what worries me so much."