A hugh peace demonstration planned for Bonn Oct. 10 has aggravated the already unpeaceful state of West German Chancellor Helmut Schmidt's Social Democratic Party.

Schmidt, describing the rally as a potential "challenge to the government," sought to discourage party participation in it during a closed-door meeting last week of the party presidium and parliamentary group. But by the week's end, 55 Socialist members of parliament -- roughly one-fourth of the party's parliamentary faction -- had signed a petition announcing they would be in the streets of this small city next weekend alongside a wide assortment of groups, including communist, environmental and church organizations, expected to swell the demonstration to well over 100,000 participants.

Schmidt's warning, moreover, appeared to clash with remarks by party chairman Willy Brant, who sounded more sympathetic to the protest. Brant cautioned against violence and one-sided complaints citing U.S. nuclear missiles while ignoring Soviet ones, but the former West German chancellor did not oppose participation in the rally by Socialist deputies.

The general purpose of the demonstration is to show concern for the buildup of nuclear weapons in Europe. But many groups participating in the action here specifically oppose Atlantic alliance plans to station U.S. medium-range missles in Western Europe beginning in late 1983.

The event, likely to be West Germany's largest antimissile protest so far, comes one month after a large and partly violent demonstration in West Berlin that coincided with a visity there by Secretary of State Alexander M. Haig Jr.

The Schmidt-Brandt clash was somewhat exaggerated by a West German press primed these days for any new trace of strain in the Socialist Party's beleaguered leadership and followed severe Socialist election losses in the state of Lower Saxony last weekend. The losses were interpreted as a sign of a national trend of public annoyance at the party's intramural strife over both military and economic security issues.

The two leaders' contrasting comments on the Bonn rally reflected the basic differences that emerged last week between Schmidt and Brandt over how best to reinvigorate the party.

Schmidt, mindful of the relatively silent majority he perceives in his party and in the country who still support his government's firm endorsement of the NATO missiles plan, takes a tough stance to win back disaffected party regulars, largely working-class types.

Brandt, reaching out for youth votes, holds to the possibility of keeping the party's left wing integrated with the rest, although this often requires skillful ambiguity in his public statements.

In an interview with a party magazine last week, Brandt again seemed to be going in two direactions at once. Appearing to endorse the peace movement, the party chief remarked, "There have been worse things on German soil than young people taking a stand for peace and disarmament." Then, changing gears lest he be accused of counseling pacifism or neutralism, Brandt said unilateral disarmament was not an acceptable course.

At a closed-door meeting here this weekend of 500 of the party's grass-roots functionaries, party leaders could do little more than attempt yet another time this year to smooth over the differences while admitting they have an image problem, to say the least.

The results of a poll, meanwhile, released by the Allensbach Institute, confirmed that the erosion in party support recorded in last weekend's state election was indeed part of a national trend.The survey showed the Social Democrats had slipped seven percentage points, to 36 percent, since last October's federal elections, while the opposition Christian Democrats had climbed to around 50 percent.

The Free Democrats, the junior partner in the Bonn coalition government, have held at about 10 percent.