BRUSSELS, OCT. 23 -- Several strange and, in retrospect, tell-tale signs of trouble had developed this morning. But not until Soviet leader Mikhail Gorbachev was well into a discussion of the expected Washington summit, in the early afternoon in an ornate hall in the Kremlin, did Secretary of State George P. Shultz and his aides begin to sense that their mission would meet a curious end.

"We had good communication in the first summit {with President Reagan in Geneva in November 1985} and good objectives and achievements in the second summit" in Reykjavik last October, Gorbachev said, according to a U.S. participant in today's meeting.

The third summit, Gorbachev went on to say, should produce some substantial achievements.

Gorbachev said he was pretty sure the two sides could agree on a treaty banning medium-range and shorter-range missiles, the signing of which had been thought to be the only major prerequisite for a Washington summit this fall. But unexpectedly, he went on to say that strategic and space arms are "the central questions" and that a summit agenda should include agreement on "key provisions" in this area.

Shultz later told reporters that at this point, "I had to say that I could not guarantee that at all. There are differences of opinion here. I said the president and all of us are quite determined we're going to find the answers to questions about defending ourselves" through the use of antimissile weapons in space.

According to a participant's notes, Shultz then said, "Some subjects have to be left for you and the president."

After further discussion, Gorbachev replied, "I guess the kind of summit I was talking about {in Washington} doesn't seem to be emerging from all this."

The surprised Shultz suggested that if this was the case, perhaps another way could be found to sign the treaty on INF, or Intermediate Nuclear Forces -- if not by Reagan and Gorbachev, perhaps by Shultz and Soviet Foreign Minister Eduard Shevardnadze, or even by the Geneva negotiators.

Gorbachev did not respond except to say he did not feel comfortable in making a summit decision at this time, according to a U.S. account, and would write a letter outlining his views to Reagan.

Shortly thereafter the meeting ended with irresolution about the signing of an INF treaty, the fate of another Reagan-Gorbachev summit and the future of U.S.-Soviet relations in the final 15 months of the Reagan administration.

"We don't know what happened. . . . It was quite surprising. We just came to this perplexing end," said one of the U.S. officials who sat through the crucial 4 1/2-hour meeting with Gorbachev in St. Catherine's Hall of the Kremlin today, and then flew later to Brussels with Shultz to brief North Atlantic Treaty Organization allies here.

Among the possible explanations being discussed among American officials in the wake of the sudden developments are "one last try" by Gorbachev at shaking Reagan's refusal to restrict his program for a space-based antimissile defense system, a Gorbachev miscalculation about Reagan's adamant stand on this issue, unseen political developments in the Soviet Union possibly connected with this week's unexplained Communist Party Central Committee meeting or the Soviet reaction to the U.S. stockmarket crash or Persian Gulf military developments.

"We just don't know," said an official who cited all these as possible explanations for the setback to the summit preparations.

The meetings in Moscow began with cordiality and cooperation yesterday, with all indications positive for an INF deal and the announcement of a date for the expected summit meeting in Washington this fall.

Shultz told reporters aboard his plane en route here that his first indication of trouble came early this morning when U.S. members of a joint "task force" on INF issues reported that their final sessions late yesterday were "totally unproductive," with Soviet participants suddenly proposing to defer decisions on outstanding issues to further negotiations in the Geneva arms talks.

Next, at 9:30 a.m., a Shultz meeting with the previously cooperative Shevardnadze "didn't seem to go very well," Shultz recalled.

Then a U.S.-Soviet task force meeting on outstanding INF issues did not happen because on the Soviet side, "nobody showed up. We were stood up," said Shultz. Another participant said that the Soviets may have been caucusing to produce answers to 20 questions the United States had asked -- and which were finally answered late in the day.

Finally at 11 a.m. came the meeting with Gorbachev, which began with good-natured greetings and banter with reporters about a Gorbachev trip to the United States. "I think it is going to happen," the Soviet leader replied to shouted questions from the U.S. press.

From the U.S. standpoint, Gorbachev's chess-board moves today breached the agreement concluded by Shultz and Shevardnadze in their Washington meetings of mid-September. A final statement issued then said:

"In order to sign a treaty on intermediate-range and shorter-range missiles and to cover the full range of issues in the relationship between the two countries, a summit between President Reagan and General Secretary Gorbachev will take place. The summit will be held in the fall of 1987, with exact dates to be determined during talks between the secretary of state and the foreign minister in Moscow in October."

Shultz as well as other officials in his party said tonight that despite the summit stalemate, progress was made in several areas of arms control, including INF and strategic arms, and productive discussions were held on a range of regional, bilateral and human rights issues. No major breakthroughs were reported.

There was keen anticipation about the letter that Gorbachev has promised Reagan on his views. Some officials suggested it might clear up outstanding problems, while others said it is likely to deepen them.

There was agreement on one thing, however, that has been demonstrated anew: Mikhail Gorbachev is a man of surprises.