The 32-year-old man arrested in connection with the assassination of Swedish prime minister Olof Palme was formerly an activist with a Swedish political group affiliated with right-wing U.S. presidential candidate Lyndon H. LaRouche Jr., but the Swedish group yesterday said it had severed ties with the man because of his strange behavior.

Victor Gunnarsson, the man held by police in Palme's Feb. 28 slaying, became associated with LaRouche's European Labor Party in December 1984, when he approached members organizing in the streets, said Michael Ericson, a party spokesman, in a statement released in Stockholm and Washington.

"In May 1985, after noticing some unbalanced features in this person, we cut all contact with him," Ericson said in the statement.

LaRouche, 63, is the leader of a tightly knit worldwide organization known for its shifting ideology and apocalyptic predictions. He lives on an estate guarded by armed sentries outside Leesburg, Va., and recently hundreds of his loyal followers moved to that area to join him.

The Swedish party (known by its initials, EAP) is one of at least six European parties in LaRouche's orbit. The statement yesterday sought to place some distance between the group and Gunnarsson and said that police officials have told members that "their investigation is not related in any way" to the party and its members.

The party said it is cooperating with the probe, but that "no member of the EAP has ever been questioned by police on the matter."

The party charged that the international media was engaging in "fraudulent, dirty propaganda" linking the group with Gunnarsson. It also alleged that the Soviets were behind Palme's assassination and are now trying to blame EAP.

A top LaRouche aide said that the EAP has 1,800 members. But Swedish press reports said the party, which started in the 1970s, is much smaller, and gained only 391 votes in legislative elections last September.

The EAP is based in Wiesbaden, West Germany, and the titular head is LaRouche's wife, Helga Zepp-LaRouche. The party and other LaRouche affiliates keep in close contact with LaRouche's Virginia headquarters.

The Swedish party has closely followed LaRouche's shifting ideology, according to interviews with former and current LaRouche followers and observers of the group, as well as an examination of various LaRouche organization publications.

The EAP -- which refers to itself as a "co-thinker organization" of LaRouche's -- supports President Reagan, the U.S. military buildup, the Strategic Defense Initiative, and nuclear power as an energy source. It is intensely anti-Soviet and anti-Palme, they said.

LaRouche publications in the United States have called Palme "a Soviet asset" and charged that a recent biography of him is a cover-up because it omitted a "missing chapter" about Palme's links to Nazis. Swedish newspapers quote EAP literature there as describing Palme as "a lunatic," "a raging beast," and "the devil incarnate."

An internal memorandum LaRouche wrote to his Swedish and other European followers several years ago accused them of being lazy in promoting his political positions and said members must realize they are suffering a "pathological deterioration" and "insanity."

Despite an often confrontational style, law enforcement authorities say the group is not now known as violent. The last serious violence associated with it occurred 13 years ago, when LaRouche ordered members to attack Communist Party members. There were several dozen clashes in this country between LaRouche's associates and Communists.

In the 1960s, LaRouche was a Marxist. He then turned to the far right in the mid-1970s. His supporters solicit funds by predicting catastrophes such as worldwide famine or nuclear disaster unless the public rallies to his cause.

Swedish authorities said anti-Palme literature, including some of the EAP's, was found in Gunnarsson's apartment. Officials said he had made threats against Palme in the past, saying "Palme is on the death list."

Gunnarsson's former wife, quoted by a Stockholm newspaper, said, "he thought Sweden would be sold to the Russians in two years."

Palme, 59, was gunned down Feb. 28 as he walked home from the movies with his wife, Lisbet.