Egypt broke diplomatic relations today with five militant Arab states that oppose President Anwar Sadat's efforts to negotiate peace with Israel.

Diplomats of the five so-called rejectionist countries - Algeria, Iraq, Libya, Syria and South Yemen - were called to the Egyptian Foreign Ministry and told to leave the country within 24 hours, according to the official Middle East New Agency.

The move came in retaliation for a decision earlier today in Tripoli, Libya, by four of the five countries to "freeze" relations with Egypt and to establish a united front to block a feared separate peace agreement between Egypt and Israel. The fifth country, Iraq, had walked the failure to adopt even harsher anti-Sadat measures.

Egypt's dramatic gesture, the latest in the series of diplomatic shocks that this country has dealt in the past month, represents the deepest split in the ranks of the Arabs in modern times. It reflects Egypt's determination to pursue its drive for a negotiated Middle East settlement no matter what the radical Arab states think.

The prevailing attitude here was deftly summed up by the headline on an editorial in today's English-language Egyptian Gazette. It called them "the mice that roared."

It appears to be a case of "You can't fire me. I quit," since the same Arab states that Egypt has now formally east aside had attempted in Tripoli to organize mass opposition to Sadat's peace initiatives.

From the first moment last month when he indicated his willingness to go to Israel. Sadat, who governs 40 million Egyptians, has shown himself determined not to be dissuaded by the likes of South Yemen, an impoverished Marxist state of 1.5 million at the end of the remote Arabian peninsula, or by Libya, which was thrashed by Sadat's armed forces in a brief border war last summer.

The decision in Tripoli of the militant countries to form a unified mutual security pact and to freeze their diplomatic relations with Egypt was predestined not to have any discernible effect on Sadat.

But Syria is another matter. Until today the Egyptian leadership had held out the hope that Syria, despite its criticism of Sadat's trip to Israel, could be brought back into the negotiating process when it really mattered, such as at Geneva.

A break in diplomatic relations was not anitcipated and will certainly make it difficult for U.S. Secretary of State Cyrus Vance to bring Syria and Egypt back onto the same track when he comes to the Middle East later this week.

Syrian reaction to Egyt's severing of relations was harsh.

Mowaffak Allaf, the Syrian ambassador to the United Nations, told UPI that it was "a most unfortunated development" and said: "While we see the president of Egypt having most cordial relations with the enemy, the people who are occupying his territory, he is breaking relations with his Arab allies."

The countries with which Egypt is severing its ties are the few reamining Arab states that have closer relations with the Soviet Union. Sadat has accused the Soviet Union of being responsible for the "rubbish" that is coming out of the conefrence at Tripoli and he had already recalled his ambassador from Moscow for consultations.

This ever deepening split between Egypt and the Soviet Union and Moscow's refusal to attend the coming Cairo conference at which Israel and Egypt are expected to discuss peace terms is believed by some observers here ot be a source of anxiety in the United States. It is thought to account for he U.S. decision to send Vance to the Middle East this week.

Egypt has so far not reacted officially to the news of the Vance trip but there is at least a faction in the Egyptian leadership that is suspicious of U.S. intentions, believing that the United States is trying to reagin the Middle East initiative from Sadat.

It is not clear what attitude Egypt intends to adopt toward the Palestine Liberation Organization in Tripoli, the PLO joined the rejectionists in their call for action against Egypt but PLO Chairman Yasser Arafat was not present when the PLO position was announced. Egypt is still hopping for an accomodation with the PLO, If not about the Cairo conference then about the question of Palestinian representation at Middle East peace talks in Geneva.

Observers here said they could not recall any previous episode in the annals on the modern Arab states in which one broke relations with so many others at one time.

Little is heard in Egypt these days about the objectives of Arab unity; on the contrary, the Egyptians are happily telling the others they can do without them, flaunting their power as the most populous Arab state.

Taking their cue from Sadat, Egyptian officials and editorial writers have been proclaiming that this country is the leader of the Arab world and is not going to take orders from states like Libya. But it had not been anticipated that Egypt would initiate any action against the others.