Although the five day Social Democratic Party convention here last week was a personal triumph for West German Chancellor Helmut Schmidt, it also reflected some signs of an uncertain future for the party that has led Western Europe's most popular nation for the last 10 years.

It was, for example, a rocky convention for Willy Brandt,the venerable party chairman and former chancellor who has given West Germany and the Social Democrats much international prestige and who is a key figure in the attempt to keep the allegiance of the left wing of the diverse party.

An attempt to articulate the strivings of the left while simultaneously endorsing the more conservative course of Chancellor Schmidt, Brandt's keynote speech satisfied nobody.

Brandt subsequently was reelected party chairman, winning 360 out of 402 votes. The number of votes against him or abstaining, however, was more than double what it was two years ago.

Some delegates from the party's left wing felt that Brandt had deserted them on the crucial issues of nuclear power and nuclear weapons. In West Berlin and other cities, such as Augsburg, there were reports over the weekend of small groups of Social Democrats renouncing their party membership.

The week's event seemed to reflect some fading of Brandt's star, which also dimmed last year when he had a heart attack and began proceedings to divorce Rut, his wife of 30 years.

Brandt's problems, along with the aging of the party's floor leader in parliament, 73-year-old Herbert Wehner -- who has guided the Social Democrats' fortunes since the postwar era began -- mark a serious challenge to the party's future leadership.

Indeed, there is talk in party circles that even Schmidt, assuming the victory he is likely to win in October's general elections, might only serve two years of a new four-year term. In this view, he would turn over the chancellorship in midterm to a younger rotege -- perhaps Defense Minister Hans Apel -- to give his successor experience and hs party a shot at holding power beyond the 1984 elections.

The Social Democrats are relatively rich in younger political talent in both the party and the circle of key aides and Cabinet ministers in Schmidt's government. But the popularity and political skill of the Schmidt-Brandt-Wehner trio will be hard to transfer in a country where the joint conservative opposition actually is larger than the Social Democratic Party.

Schmidt's party rules in coalition with the smaller Free Democratic Party but there, too, questions loom over future leadership. Party Chairman Hans-Dietrich Genscher is also Bonn's foreign minister, a double assignment that is fraught with stress and probably has contributed to Genscher's two hospital stays for heart problems in the past year.

The Social Democrats support here -- at Schmidt's urging -- of the limited use of nuclear power to provide Bonn's energy and long-term economic security also entails serious political risks because of the growth of a new environmental, or "green," political party.

Although thus far they are very small, and composed of disparate elements -- including conservative conservationists and Marxists -- that ultimately could cause their downfall, te "Greens" nevertheless are a threat because of electoral laws that require parties to get at least 5 percent of the vote to be represented in parliament.

It is felt that the environmentalists could take enough votes away from the Free Democrats, who only got 7.9 percent inthe last federal elections, toknock them out of parliament and thus unseat the ruling coalition.

Thus, Schmidt and Brandt are tryingto emphasize their own party's concern for the environment while telling people that a vote for the "Green" is, in effect, a vote for the ultraconservative challenger for the chancellorship, Franz Josef Strauss.

Soon after the party assembly here closed, Strauss told reporters in Munich that he was honored at being the featured attraction at the Social Demcrats' convention.

Strauss, of course, was joking. But the extraordinary attention he got here in speeches suggests that the Social Democrats are perhaps not quite as confident of defeating him as they might otherwise seem.

Strauss was the main focus of Brandt's speech, which contained a level of invective that struck many observers as the harshest ever at such a gathering.

Brandt said Strauss' "uncontrollable and unpredictable energy" had come to symbolize a type of German who "uses his elbows to get ahead." He said all the postwar good will won by West Germany around the world would be destroyed by a Strauss victory, which would provoke "new fear of Germany, and in Germany."

Thus, while Helmut Schmidt's position as the leader of his party and a dominant figure in Western Europe currently is unchallenged, his fellow Social Democrats have signaled a bitter election struggle in 1980. CAPTION: Picture, HELMUT SCHMIDT . . . faces bitter election fight