Karl Heinz Hoffmann seemed hardly worth making a fuss about. At least that's what the West German authorities said for seven years.

Hoffmann and his thuggish band of neo-Nazis brandished guns, engaged in paramilitary maneuvers, honored Adolph Hilter and cursed the republic in which they live. Gradually, their membership spread like ooze from the southern territory of Bavaria into three states. But police considered the group more deranged than dangerous.

In fact, there wasn't much that distinguished the Hoffmann corps from the several dozen other small right-wing packs in modern West Germany, closely monitored by the government but largely dismissed as misfits. Except for one thing. Hoffmann, a 43-year-old former mercenary and sign painter with a handlebar mustache you could almost chin yourself on, relished publicity.

It was press attention that ultimately turned Hoffmann into a threat to the highly image-conscious Bonn government. This neo-Nazis had become on annoying public relations blot. That is why Interior Minister Gerhart Baum last month took the unusual step of banning Hoffmann and his Wehrsportgruppe (literally, military sports group).

Rarely has the West German government gone so far as to outlaw neo-Nazi or other extremist groups. Unless a criminal case can be built, officials here say, they have difficulty finding cause for action and are reluctant to exercise government power over constitutionally protected freedoms of assembly.

"There are two ways to get rid of such groups," says Hans Joachim Frieling, an official of Bavaria's Interior Ministry. "One is by government action. The other is death by silence.My opinion is that none would take notice of these groups if it weren't for the press."

But the Hoffmann case developed into a public embarrassment precisely because it was singled out by both the domestic and foreign press as an example of state inaction.

Just last month, West German television aired a commentary quoting officials disclaiming responsibility for Hoffmann's troopers. Bavarian authorities said that although Hoffmann had his headquarters in their state, he had expanded his activities into other jurisdictions and so had become a matter for the Bonn government. Bonn authorities responded that punishment for weapons offenses and the like rested with state and local officials. And a spokesman for the federal supreme court said that unless there was proff that Hoffmann and his band were planning crimes, no investigation could be ordered.

When Bonn finally acted, it was on grounds that the Hoffmann organization espoused anti-constitutional goals and sought to establish a totalitarian state by force. Interior Minister Baum declared that the ban against Hoffman should be a signal to other extremist groups that the government was able and prepared to prohibit them.

But it was evident that the order was founded more on political than legal or security considerations. "Because of the sufferings caused by National Socialism," Baum said in a statement, "the activities of these groups are especially observed in foreign countries and mean a great burden to the reputation of the Federal Republic which cannot be tolerated any longer."

Gerold Tandler, the interior minister of Bavaria, echoed this theme in his public comments on the case. "It is clear that other groups are more dangerous," he said, "but Hoffmann's specialty was to damage the international reputation of the Federal Republic."

Organized right-wing extremism is hardly a significant political factor in West Germany today. Indeed, the total known membership of the organized ultra-rightwing groups has been falling steadily in the past 10 years.

Most of this decline, however, has been accounted for by the decay of the National Democratic Party after its moment of glory in the late 1960s when it won seats in several state parliaments. The NDP is the closet thing today to Hitler's party, and it is the only party of the extreme right to have been on postwar German ballot. Its membership list of mostly older, discreet ultra-nationalists now totals something under 8,500.

What worries authorities is not this relatively weak ultra-right segment, but the growth of an increasingly violent neo-Nazi element. Baum disclosed that the number of these hard-core extremists had jumped last year from 400 to 1,400. Last year more than 1,000 neo-Nazi offenses were recorded -- such as desecration of Jewish graves, disturbing the peace, painting racist slogans -- compared with 319 in 1976.

The "new right" can hardly be described as a movement. It is scattered, disorganized, lacking a common strategy or leaders who appear capable of commanding a wide following. Counting all extremist rightwing organization, not just strictly neo-Nazis, the tally at the end of last year was 69, with a total membership of about 17,300 -- down about 300 from 1978 and a tiny sum in a nation of more than 62 million people.

Who are these people? Police say most neo-Nazis are young, between 14 and 25, with a few older ones who serve as a bridge to the dwindling Hitlerites of an earlier era. Unlike the extremists on the left, the New Right are mostly nonintellectual, craftsman, often unemployed, with a desire for action and, according to one state police study, demonstrating "a rage against anything existing."

West German law stricly forbids the distribution of Nazi literature and the wearing of Nazi insignia, although authorities have been known to turn a blind eye to certain kinds of Nazi-style behavior.

Police reject that charge, pointing to a steady rise in arrests and court cases involving the most serious offenses. It is their claim the neo-Nazis easily manipulate the press, getting magazines and television to pay for permission to photograph meetings and demonstrations. Bavarian police say that Hoffmann was particularly clever in this regard. As a self-styled proponent of authoritarianism, he was suitably outrageous. He and his 400 followers carried all the right trappings.

In a predawn raid last month on Hoffmann's home in the little town of Heroldsberg, seven miles northeast of Nuremberg, and on 22 other sites connected with the neo-Nazi group, police confiscated rifles, pistols, bayonets, gas masks, camouflage suits and a bust of Hitler. In Hoffmann's basement, police discovered a puma that was handed over to the Nuremberg zoo. At a ruined castle at Erlangen, 11 miles north of Nuremberg, where Hoffmann's troops periodically conducted paramilitary exercises, authorities seized an anti-craft gun, a 12-ton armored car, troop trucks and a West German army jeep.

Hoffmann himself was not arrested. Federal officials said that it would be left to local authorities to pursue charges for weapons and other violations. Hoffmann was a given a suspended jail sentence in 1977 following a brawl with left-wing students at Tuebinger University. He received another suspended jail term last year for wearing a Nazi-style uniform at a rightwing rally.

Hoffmann's neighbors were somewhat surprised by all the commotion on their street. "He and his people were always very polite," one neighbor told a West German news agency. "They never drank too much." Said another: "He took a lot of young people off the street, who before just loafed about."