Argentina and Chile signed a protocol at the Vatican today that is expected to end a century-old border dispute over land and sea rights at the tip of South America. The issue nearly caused a war in 1978.

The tentative accord on the Beagle Channel dispute, the fruit of a five-year mediation effort by Pope John Paul II, was signed by diplomats from the two countries in a ceremony hosted by the Holy See's secretary of state, Cardinal Agostino Casaroli.

The pope began seeking a solution of the dispute between the two largely Catholic countries two months after his investiture, following Argentina's near invasion of Chile on Dec. 23, 1978.

A few hours after the signing, The Associated Press reported, Chile accused Argentina of firing eight cannon rounds at a lighthouse across the Beagle Channel, with the shells falling harmlessly in the water. Argentina denied the charge.

In Buenos Aires, Foreign Minister Dante Caputo hailed the pact as a "triumph of reason, diplomacy and peace." He said the text of the proposed treaty would be released Friday. The protocol says the two countries intend to accept the treaty, which must be ratified by both governments.

Caputo said the pact reflected concessions by the two countries and was "a balanced agreement that satisfies the interests of both sides."

President Raul Alfonsin's government, burdened with acute economic problems, is known to have wanted to reach a settlement on the issue in order to halt arms purchases and to promote needed regional economic integration. Officials also say that a compromise by the Argentines on the Beagle issue will help them in their continuing dispute with Britain over possession of the Falklands.

Alfonsin had made signing an agreement a key issue in last year's presidential race. The Beagle dispute long has centered around the ownership of three small islands -- Lennox, Picton and Nueva -- and the surrounding territorial waters in and around the Beagle Channel -- named after the ship that British naturalist Charles Darwin used to navigate the waterway in 1832.

Although Chile's position long has appeared to be favored by international law, the militarily stronger Argentina had held out against any concessions. A 1976 arbitration put before the British crown would have resulted in award of the islands to Chile, but it was rejected by Argentina.

By December 1978, tensions had reached a flash point. One Argentine general reportedly had glasses etched "Santiago 1978" for a victory celebration, which was not to be, in the Chilean capital, and on Dec. 23, Argentine troops in forward positions began to move across the border -- only to be called back at the last moment.

Diplomats here say the agreement signed in Rome is a compromise, with Chile winning the islands and a promise by the Argentines to accept binding arbitration should problems develop in the region.

The Argentines are said to have gotten a 1980 papal proposal for a 47,000-square-mile "sea of peace," or mutual economic exploitation zone, replaced with Argentine control over that Atlantic area. It reportedly also won jurisdiction over the eastern mouth of the Strait of Magellan, to the north.

Diplomats here say that although the negotiating work on the agreement had been nearly completed before Argentina's military rulers left office Dec. 10, pressure from right-wing nationalist officers had prevented an accord.

Last January, the foreign ministers of the two countries took a major public step by signing a declaration at the Vatican pledging to work out a treaty.

In July, Alfonsin called on Argentines to participate in a nonbinding plebiscite prior to his final recommendation to the Congress, which must ratify the pact.

Calling of the referendum appears to have neutralized opposition. The powerful Argentine Catholic Church is backing the treaty. One bishop said that to vote "no" in the referendum next month meant boting against the Pope.