MADRID, OCT. 7 -- The Guatemalan government opened peace talks with leftist rebel leaders today in the first formal effort to negotiate an end to the Central American country's 26-year-old guerrilla war.

The meeting here, although described as exploratory, ushered Guatemala into an expanding pattern of talks to stop the conflicts of Central America in accordance with a regional peace plan signed Aug. 7 by the leaders of Nicaragua, El Salvador, Guatemala, Honduras and Costa Rica.

President Reagan and his aides, focusing primarily on Nicaragua, initially disparaged the Costa Rican-sponsored peace plan as fatally flawed. Central American governments nevertheless have taken the first tentative steps toward its goals of cease-fires and negotiations on democratic reforms and an end to outside aid or refuge for insurgent forces.

The Guatemalan conflict, which began in 1961, is the oldest and in some ways the most difficult of Central America's guerrilla wars to settle in negotiations. With consistent and brutal attacks and large-scale resettlement of the population, the Guatemalan Army has significantly weakened the guerrilla alliance, the Guatemalan National Revolutionary Union, in all but the country's most remote areas.

The number of Guatemalan guerrillas has shrunk to about 2,000 from a high of more than 10,000 at the beginning of the decade, according to Guatemalan estimates. Army officers, who retain great authority in Guatemala, have expressed reservations about any peace talks in view of their military success and the election in 1985 of the country's Christian Democratic president, Vinicio Cerezo.

The guerrilla leaders, whom Cerezo has urged to join legal politics, remain suspicious of the Army's declared willingness to allow dissent, citing the country's record of political murders and disappearances. Reports from Guatemala said the discussions were being held in Madrid in part because rebel leaders feared for their lives if they returned openly to Guatemala City.

Cerezo's government, sensitive to hesitations within the military, has referred to the discussions as "contacts." Guatemalan Ambassador Danilo Barillas told reporters both sides have agreed to keep details of the talks secret.

A joint communique late tonight said the talks began this afternoon but did not disclose the site or the substance of the discussions.

Although secret contacts are believed to have been held between guerrillas and envoys from Cerezo, the discussions here were described as the first organized peace talks since guerrilla warfare began. Observers said that although the discussions were unlikely to produce swift results, it was significant they were being held at all in the light of Guatemala's long years of war without quarter by both sides.

The guerrillas declared a cease-fire when they announced the talks last week, but a Guatemalan official in Washington said the government had not formally announced a truce.

Prime Minister Felipe Gonzalez's Spanish government acted as host but avoided a substantial role in the talks, Spanish officials said.

The communique said the Guatemalan delegation was headed by Roberto Valle, first president of the Congress. He was accompanied by four military officers.

The communique said the head of the rebel delegation is Rodrigo Asturias, whose name in the guerrilla organization is Gaspar Ilmo.

Asturias is the son of Miguel Angel Asturias, the Nobel Prize-winning Guatemalan writer, and head of the Organization of Armed People, one of three guerrilla groups in the alliance.

The other two are the Rebel Armed Forces, headed by Pablo Monsanto, and the Guerrilla Army of the Poor, headed by Rolando Moran.

The Guatemalan Workers' Party, a communist organization, has withdrawn from the alliance but, according to reports from the area, still supports it actively.