MADRID, SEPT. 16 -- After nearly a year of negotiations, Spain and the United States remain at loggerheads over Spanish insistence on "substantial" reductions on the 12,000-strong U.S. troop presence in this country.

One more negotiating session is scheduled for October, before a mid-November deadline by which the Madrid government must officially inform Washington of an intention not to renew the current five-year basing agreement. It expires next May.

The Spanish already have signaled their intention to send the required diplomatic letter. If no accord is reached by May, the Americans have until May 1989 to evacuate their three air installations and one naval base here and go home.

Both governments insist that they do not want this to happen and that they are committed to maintaining the 34-year-old bilateral security accord. But judging by a series of conversations here this week with observers on both sides of the talks, Madrid and Washington appear to be locked into rigid positions.

Madrid presents restructuring of the agreement as final proof, 12 years after the death of dictator Francisco Franco and the installation of an elected government, that it is a bona fide democratic member of the North Atlantic Treaty Organization and capable of making its own decisions, while safeguarding alliance interests.

While the Madrid side says it is insulted by the U.S. refusal to accept its wishes, the Americans say Spain's reluctance to see that U.S. forces here are vital to western security is evidence that Madrid is playing domestic politics and still may lack the maturity of a full-fledged ally.

The situation is especially worrying for the United States because American capitulation to Spanish demands, or Spain's cancellation of the agreement, could have negative repercussions in other countries where the future of U.S. military installations has been challenged.

The Greek government of socialist Prime Minister Andreas Papandreou periodically threatens expulsion of American forces and an unstable political situation in the Philippines has cast into doubt the future of U.S. air and naval bases there.

Last week, Portugal's conservative government protested reductions in U.S. military aid and threatened to call for a renegotiation of a treaty that gives the United States use of Lajes Air Base in the Azores islands.

Before this new threat arose, Portugal had been mentioned as a possible fallback base for 72 F16 fighters that Spain wants removed from its territory.

These fighters of the U.S. 401st Tactical Air Wing at Torrejon Air Base are the crux of disagreement between Spain and the United States. Under NATO redeployment plans, the planes would be sent to Italy and Turkey in times of crisis.

When negotiations began last October, Madrid demanded reductions at all four U.S. installations here, Torrejon, air facilities at Moron and Zaragoza and the major U.S. naval base at Rota, according to Spanish officials.

"We knew that was unrealistic," one defense official said, "but we thought that you had to start tough with the Americans so you could end up getting some of what you wanted."

Eventually, Spanish demands came down to the removal of the 401st and its personnel of up to 4,000. According to sources here, the Spanish are willing to stretch out the departure timetable for as long as five years. They have additionally pledged that Torrejon will continue to serve as a redeployment supply depot and be available to NATO forces in the event of hostilities.

The base, just west of Madrid, is a visible reminder of the U.S. presence that began here when Franco and the Eisenhower administration signed the first of a series of five-year agreements that Spain maintains have been unfairly weighted in favor of the United States.

Franco died in 1975 and was replaced by a series of civilian governments, leading to the 1982 election of Prime Minister Felipe Gonzalez and his Socialist Workers' Party.

Gonzalez's election came just months after his centrist predecessor had signed a new five-year agreement and had decided to take Spain into NATO, a move that public opinion polls showed was unpopular.

Gonzalez had campaigned on a promise to freeze entry into NATO pending a referendum on the issue. But by the time it was held in March 1986, he had ensured approval of membership by including provisions barring Spain's incorporation into NATO's "integrated military structure" and pledging reduction of U.S. forces.

Since passage of the referendum, Spain has been negotiating with NATO administrators over a new kind of alliance membership that Defense Minister Narcis Serra last week called the "Spanish model." Speaking to the International Institute of Strategic Studies' annual conference, held in Barcelona, Serra said Spain would not integrate its troops into any non-Spanish NATO command, nor would it allot its own military officers to NATO regional commands.

But Spain would participate in NATO, he said, by defining and coordinating its own defense planning structure along the same lines as that of "integrated" NATO countries and would "establish coordinating agreements to specify what missions would be done under Spanish command" in a crisis.

Serra said Spain saw its primary mission within NATO as "safeguarding air and sea routes" around and across Spanish territory for U.S. resupply of its allies farther to the east.