A cavernous beer hall in this Bavarian town, given to good brew and loud choruses, echoed with soft classical tunes one recent evening and its dark walls carried bright sunflowered posters, trademarks of West Germany's unconventional Green Party.

At a small podium, a retired West German general, Gert Bastian, talked excitedly about how Western security policy had reached a "dead-end street."

Off in a corner, where the mugs and dishes normally are stacked, Green Party organizers displayed booklets and buttons describing protests against atomic weapons, nuclear power plants, polluters of nature and centralized decision making.

Even in this southern corner of the country, a farming region of die-hard conservatives, the Greens, as they are called, managed to draw a big roomful.

Dismissible a year or even six months ago as a ragtag assortment of obstructionists, extremists and idealists, this radical party is now a definite political force in West Germany.

Its aim--to the extent one can be distilled from the party's diverse groups--is to transform West Germany from Western Europe's most powerfully militarized, industrialized, bureaucratized state into a nuclear-free zone chock-full of pollution controls, a decentralized country where bureaucratic power would be rechanneled to individuals and local councils.

The Greens argue for a new West German direction, away from the United States, capitalism and other post-World War II standards, but not necessarily toward communism or socialism.

"The Greens are not left or right," said Petra Kelly, the party's most widely recognized national spokesman, in an interview as she stopped here to campaign for a state parliamentary seat in October elections. "We're trying to find a third way, a self-determining third way between capitalism and socialism."

Kelly called this "a new form of German patriotism."

Since organizing in 1979, the Greens in one form or another have won seats in the parliaments of four of 10 West German states, including Bremen, Baden-Wuerttemberg, Lower Saxony and Hamburg. They also are in the Berlin Senate and smaller local councils.

Opinion surveys this summer showed the Greens passing the centrist Free Democratic Party in national popularity. The Free Democrats have ranked third since the start of the federal republic 33 years ago, acting as a key pivot at times to hand power to either the Social Democrats or Christian Democrats by forming coalitions.

In the state of Hesse, where Sept. 26 elections are considered immensely important for the stability of the Bonn government, the Greens are showing double the strength of the Free Democrats, with 11 percent in independent polling.

The rise of the Greens reflects deepening voter disenchantment with West Germany's traditional three parties. Months of squabbling and divisions, plus investigations into a party payments scandal, have eroded public support for the 13-year-old Bonn coalition of Social Democrats and Free Democrats.

The conservative Christian Democrats have gained some strength. But their leadership often is seen as lackluster and unattractive to the disaffected abandoning the ranks of the ruling parties and to many young people voting for the first time.

What the Greens offer is something outside the old German consensus. They profit from an image of creativity, dynamism and imagination growing out of what some consider their emotional intensity and lack of clarity or organization.

They are an outgrowth, too, of heightened German anxieties about war, nuclear attack, economic collapse and technological catastrophies. Their call for grass-roots democracy answers a sense of alienation among many young people and appeals to Germans who, in increasing numbers, have joined citizens' protests.

Although the party's demands appear impractical to some, the appeal is particularly understandable in a nation in which the romantic and the abstract have traditionally coexisted in a certain tension with the concrete.

The Greens are stubborn. They have refused to compromise with other parties on the two issues on which all Greens can agree: no nuclear weapons on West German soil and no nuclear power plants. Refusing so far to enter into any coalition government, they have introduced a new phrase to political discourse here--describing their parliamentary role as that of a "fundamental opposition" party.

This stance has infuriated the conventional parties. In a recent multiparty television discussion, Kelly was sharply attacked for expecting that the Greens be accepted as legitimate when they refuse to play by the standard political rules.

Nonviolence, too, is a rule some Greens do not look at in the conventional way. While ruling out violence against people, not all party leaders have excluded violence against things, such as the controversial new Frankfurt Airport runway extension. Nor did the leader of the Greens' parliamentary faction in Hamburg consider it wrong to occupy a police station recently.

This sort of attitude encouraged a leading Social Democrat to suggest the Greens are fascists. "A political group that has declared war on parliamentary principles," said Hesse's minister-president, Holger Boerner, last month, "has learned nothing from German history but is close to fascism." Boerner is facing a strong Green challenge in a tough reelection fight this month.

Chancellor Helmut Schmidt sounded a little more tolerant. He told a newspaper interviewer the Greens reflect a "fermenting process" that may or may not lead to maturity. At the moment, Schmidt said, the Greens are still "more a movement of protest on every imaginable issue than a party."

Even so, the chancellor's Social Democratic Party, in a clear acknowledgment of the Greens' new clout, has been forced in Schmidt's home town of Hamburg to begin cautious exploratory discussions on working out some cooperative arrangement--short of a full-fledged coalition--that would permit the Social Democrats to carry on a minority government. Elections in Hamburg last June left both the Social Democrats and Christian Democrats shy of a majority. The Greens hold the remaining nine seats in the city-state parliament there.

Greens spokesmen readily concede their party has to grow up, which they say is a principal reason for not compromising now. By shunning trade-offs with the established parties, the Greens hope to affirm their credibility as an alternative party.

"You can be responsible in government without sharing power," said the U.S.-educated Kelly, 34, who has a job in Brussels advising the European Community on social and economic policy. "I don't think we're ready yet. We have to learn unconventional ways first to get at the answers to things."

Just who are the Greens? A poll by the Allensbach Service reported last month that about two-thirds of the party's supporters are under 30, a majority of these still students. But a large percentage of Greens also come from teaching and civil service.

Joining disgruntled ex-Social Democrats and committed ecologists are former or active Communists, anarchists and other representatives from the fringes.

Contrary to the "dropout" tag applied to them, Allensbach said the Greens tend to be exemplary for the political energy and involvement they show.

Curiously, though, for a party that has benefited from a national atmosphere of anxiety, the Greens do not seem to fear all the things other West Germans are likely to -- such as, according to Allensbach, a Soviet attack, future energy shortages, a rise in crime, or political extremists holding civil-service jobs.

When asked whether West Germany should "cooperate equally closely with America and Russia or more closely with America or more closely with Russia," 72 percent of the Greens polled said equally.

Where Greens have been elected, they have behaved in different ways. In the traditionally conservative state of Baden-Wuerttemberg, where the party drew its strength initially in 1980 as a movement of farmers and local residents against an atomic reactor at Wyhl, the Greens have blended into parliament. In Bremen, they remain sharply obstructionist.

The party's 10-member Executive Committee still often appears split over strategy and policy. Recently a clash erupted when one committeeman, Roland Vogt, went to Tripoli with several other leading party members to meet with Libyan Col. Muammar Qaddafi. Other Executive Committee members later rebuked Vogt, calling the trip "politically questionable" and saying it damaged the party's credibility in foreign affairs. They suggested Vogt resign. He didn't.