West Germany's Social Democratic Party, rallying support that surpassed even its own most optimistic projections, made an impressive showing in a critical state election today that dealt a serious blow to attempts in the national parliament to oust Chancellor Helmut Schmidt.

Both Schmidt's Social Democrats and the opposition Christian Democrats lost one seat in elections for the legislature of the central state of Hesse. The Social Democrats had expected to lose some support, but the Christian Democrats had hoped to gain enough seats to come to power with an absolute majority.

In the biggest surprise, the Free Democrats of Hans-Dietrich Genscher lost badly, drawing only 3.1 percent -- the worst defeat for the small centrist party in its history. The Green Party, which ran on a platform opposing nuclear power and advocating unilateral disarmament, received enough votes to gain seats in the state assembly for the first time and now holds the balance of power there.

The Free Democrats, who had been the junior partners in Schmidt's federal coalition until nine days ago, lose representation in the state assembly because they fell below the minimum 5 percent of the vote required by the constitution.

The election outcome, described by television commentators as "astounding" and "a political sensation," deepened doubts that the Christian Democrats and Free Democrats could proceed with their parliamentary plan this week to unseat Schmidt and install a new center-right coalition in Bonn headed by conservative Helmut Kohl.

Social Democratic Party officials heralded the Hesse result as a clear vote of confidence in Schmidt and in his call for new national elections before the end of the year. In contrast, the opposition parties have proposed forming a new Bonn government before arranging for new elections next March.

Claus Boelling, Schmidt's chief spokesman, said tonight that the chancellor saw the Hesse vote as confirmation that no democratic or moral justification exists for a change in national government. There should be "new elections in any case and no more manipulations," Boelling quoted Schmidt as saying.

But Kohl and Genscher, while expressing disappointment with the vote, said they would resume negotiations for a new coalition.

The leadership committees of West Germany's opposition parties are scheduled to meet Monday to review the election results and decide how to proceed. They have until Tuesday at midnight to file a motion in the Bundestag (the lower house of parliament), for what is called a constructive vote of no confidence against Schmidt, in order to be able to hold the vote Friday as planned.

"We have experienced a disappointment," Kohl told a television interviewer, commenting about Hesse. "We have not reached our goal."

But asked if the Hesse vote changed the basis for challenging Schmidt, Kohl insisted that high national unemployment and mounting government debt constituted "a kind of state emergency" that still demanded the rapid establishment of a new government that could command a majority in parliament. Schmidt now heads a Social Democratic minority government.

Social Democratic Party leader Willy Brandt, buoyed by the election results, said tonight: "I am not certain a no-confidence motion will be introduced and if it is I am not certain it will pass."

According to the final official tabulation, the Christian Democrats scored 45.6 percent in Hesse, down from 46 percent in the last state election there in 1978. The Social Democrats received 42.8 percent, down from 44.3 percent four years ago.

After losing one seat apiece, the Christian Democrats will have 52 seats in the new parliament, and the Social Democrats will have 49.

Until last week, opinion surveys had indicated that Christian Democratic leader Alfred Dregger would gain an absolute majority in the state and usher in a new era of conservative rule. The Social Democrats have controlled the state government for 35 years.

The remaining nine seats in the Hesse assembly will be held by the Green Party, which drew 8 percent of the vote. The Free Democrats, with 3.1 percent, were down from 6.6 percent in 1978.

The Greens, who had gained seats previously in five of West Germany's 11 state parliaments, benefited in Hesse from local opposition to several major development projects -- including a runway extension of Frankfurt's Rhein-Main Airport, a nuclear fuel reprocessing plant and another nuclear power facility.

This marks the second case in West Germany in which the two major parties, denied a majority, face the prospect of trying to build a government with the Greens, who up to now have refused to enter a coalition. The other example is Hamburg, which voted in June and where the Social Democrats continue to operate a minority government. The Social Democrats in Hesse also are expected to continue to administer the state for several months at least.

Both instances suggest that new elections could produce a similar impasse at the national level. This has prompted speculation that a "grand coalition" between the Christian Democrats and Social Democrats could result eventually in Bonn similar to the one that ruled West Germany from 1966 to 1969.

"It is possible that in Germany we will have to get used to the idea that it has become more difficult to form majorities," said Peter Glotz, the Social Democratic national party manager.

Today's vote reflected a state campaign that turned highly emotional following the breakup in Bonn Sept. 17 of the 13-year coalition between the Social Democrats and Free Democrats.

In a strong publicity drive launched immediately after the split, the Social Democrats portrayed the Free Democrats as traitors and political manipulators. They argued that a new conservative government, in Hesse or in Bonn, would produce harmful economic and social effects.

Presaging the collapse in Bonn, the Free Democrats in Hesse had decided in June to seek a new coalition with the Christian Democrats, ending a 12-year partnership with the Social Democrats in the state. The move was controversial at the time, but Genscher's break with Schmidt opened wide the rift in the Free Democratic Party.