In retaliation for an arms agreement between the United States and Somalia, Ethiopia has mounted a diplomatic offensive to isolate Somalia, its traditional enemy in the Horn of Africa.

The offensive could hamper Washington's efforts to beef up its military capability in the Indian Ocean, and so might offer an early test of President-elect Ronald Reagan's policy on Africa.

Ethiopia's leader, Lt. Col. Mengistu Haile Mariam, scored a major diplomatic success in Kenya earlier this month, when he issued with Kenyan president Daniel arap Moi a communique calling for Ethiopia and Kenya to "coordinate their activities in the struggle against Somalia." Mengistu and Moi went on to accuse Somalia of adopting expansionist, "hostile policies."

Somalia, which borders on both Kenya and Ethiopia, reacted sharply, asserting that the communique was "tantamount to a declaration of war." Somali President Mohammed Siad Barre ordered a general mobilization, but his position was weakened with Soviet-backed, Marxist Ethiopia and Western-oriented, capitalist Kenya formally joining forces against his country.

There is even some concern among Western analysts that Siad Barre could be overthrown. As one Western diplomat asked, "If Siad goes down the drain, what happens to the U.S. access agreement?" He was referring to the deal under which Somalia will allow American military forces to use ports and bases in return for $40 million in arms.

Obtaining access to the air and sea facilities of Berbera, on Somalia's north coast, and of Mogadishu, in the southeast, is part of the U.S. government's policy of strengthening its ability to counter Soviet moves in the Indian Ocean and the volatile Persian Gulf area. Agreements have also been signed with Oman and Kenya, but the Somali deal, signed in August, is the most controversial, because of Somalia's claims on the territory of its neighbors in the Horn of Africa.

The U.S.-Somali agreement has not been implemented yet, since Congress has barred the United States from providing Somalia with military hardware until there is "verified assurance" that Somali troops have been withdrawn from the ogaden. For a century, Somalis have claimed the barren Ethiopian territory of the Ogaden, and during the last two decades it has been the scene of two wars and numerous skirmishes. U.S. officials believe there are stilll Somali troops in the contested area.

The removal of Siad Barre or even the weakening of his government could bring about a quick test of the incoming Reagan administration's Africa policy.

More than President Carter ever did, Ronald Reagan appears to regard Africa primarily in terms of East-West issues. In such a view, a Somalia friendly with the United States would be important in countering the expansion of Soviet influence through Ethiopia, to which Moscow has supplied more than $1 billion in weapons.

Such a view could also lead to trouble. One Western diplomat, noting recent Ethiopian victories in the Ogaden, said, "Siad has lost his war." The diplomat warned that "the more the United States does in Somalia, the more problems the Americans will have with Kenya."

Most observers feel Siad Barre is safe as long as he maintains the support of the military, which put him in power in 1969.That backing could weaken, however, if Ethiopian forces threaten Somalia itself. Ethiopian soldiers have crossed the frontier several times, but a full-scale invasion appears unlikely, as it could shift African sympathies from the Ethiopians to the Somalis.

Even so, the Ethiopian ambassador toKenya, Mengistu Desta, left the question open in an interview last week. "We've had enough," he said. "You can't antagonize a neighbor without expecting retaliation."

For Kenya, the warming of relations with its Marxist neighbor is part of a drive to improve ties with nations on its northern borders as a way of countering problems on its other frontiers with Somalia, Uganda and Tanzania.

For Somalia, the Ethiopian-Kenyan communique represents a serious setback in its efforts to lay to rest its border problems with Kenya and concentrate on the Ethiopian frontier. Britain left the matter of the Somali-Kenyan border unsettled when it granted independenced to Kenya in 1963, despite a survey of Northeastern Kenya -- heavily populated by ethnic Somalis -- showing that most inhabitants wanted to be part of Somalia. A two-year war followed, based on Somali claims to move than a third of Kenya. There has been sporadic fighting since.

Ethiopia has recently mended relations witwh another neighbor, Sudan, which lies to its north and west. The move is expected to make it harder for Eritrean guerrillas, waging an 18-year-old war for the independence of Ethiopia's northernmost province, to operate from Sudan. Ethiopian forces, backed by Cuban troops, drove the Eritreans out of most of the heavily populalted areas two years ago, but have failed to defeat them in the mountainous hinterlands.