An important regional faction of West Germany's Free Democratic Party decided today to abandon its 12-year-old alliance with the ruling Social Democratic Party, adding to speculation here that the beleaguered coalition government of Chancellor Helmut Schmidt is near collapse.

The Free Democrats in the key central state of Hesse--which includes the populous financial center Frankfurt--announced that they would seek a coalition with the conservative Christian Democrats after state elections Sept. 26.

The decision, taken in a 169-to-129 vote at a special congress of regional delegates in Darmstadt, is a major blow to Schmidt's rapidly deteriorating political fortunes.

The Hesse party's move is likely to increase pressure on Foreign Minister Hans-Dietrich Genscher, national chairman of the national Free Democrats, who is reported to be seriously weighing a switch of his party to the conservatives after a series of setbacks for the Social Democrats and Free Democrats in local and state elections.

The switch in Hesse by the Free Democrats, who have held the balance between the two larger parties on the national level for the last decade, followed the party's failure in June 6 elections in Hamburg to win enough votes to remain in the city-state parliament there. That election also saw the Socialists slip markedly in a stunning setback in Schmidt's home city, which had been a Social Democratic stronghold for years and where he personally campaigned hard.

This loss, coupled with recent defeats in West Berlin, Lower Saxony and Schleswig-Holstein have confirmed a substantial ebb in public support for the chancellor's party, reinforcing doubts among many Free Democrats about future alliances with the Social Democrats and raising questions about whether Schmidt's government will survive its full term. Federal elections are not due until 1984.

In a passionate debate preceding today's vote, delegates opposing the switch argued that their party risked committing suicide by lining up with the conservatives. They said the change would be perceived by voters as opportunistic and unprincipled. They also warned of shock waves the action would have in Bonn.

But the Free Democratic leader in Hesse, Ekkehard Gries, argued that the switch was necessary to prevent the conservatives from gaining an absolute majority.

This is important because a conservative victory in Hesse would give the Christian Democrats a two-thirds majority in the Bundesrat, the upper house of the national parliament, enough to block any legislation. To head off such power, the Free Democratic faction in Hesse is stipulating that the Hesse conservatives agree to support the Bonn coalition parties in the upper house. It is uncertain whether the Christian Democrats would accept such terms.

Gries also said that by switching, his party should be able to stop the increasingly popular environmentalist Green Party, from winning enough votes to conduct "obstructionist politics" in the Hesse parliament.

Declaring that a general change in West Germany's "political climate and party landscape" had taken place, Gries added: "We must accept that at the moment the Social Democrats and Free Democrats are not getting majorities. A great number of our members and voters obviously want a change."

Speculation, meanwhile, about the imminent demise of the Bonn coalition almost has reached a fever pitch. Twice this week, government spokesmen have been prompted to deny reports that Schmidt is thinking of resigning.

The attention is on Genscher, who is under pressure to align the party with the conservatives.

"Genscher's Hamlet dilemma: to jump or not?" teased the headline of an article in this week's Die Zeit newspaper by Ralf Dahrendorf, head of the London School of Economics and a close observer of West German politics.

The paper's co-editor, Theo Sommer, said in another article that Genscher might indeed be doing the nation a favor if he chose to switch now. Sommer suggested that an abrupt change in government might be preferable to a slow death of the coalition.

Coalition talks that began this week on the 1983 budget are seen as providing a convenient trigger for Genscher to switch if he chooses to.

A meeting between leaders of both parties Wednesday succeeded in setting a spending limit of about $104 billion dollars for next year, about 2.2 percent over this year. But a major row is anticipated as the Free Democrats push for cuts in social programs favored by the Social Democrats.

Some Social Democratic deputies are saying privately that they would prefer to push the Free Democrats into jumping now rather than try to keep the coalition loosely patched until 1984. Their worry is that if current coalition strains continue, the Christian Democrats stand to gain in all the regional balloting and could win an absolute majority in the regularly scheduled national election two years from now.

These Social Democrats argue that a retreat into the opposition might allow the party a chance to regenerate in time to mount a stronger challenge in 1984.