Libya decided yesterday to recall four members of its diplomatic mission here who had resisted a U.S. explusion order, ending a four-day standoff that included a Libyan threat to embargo oil and an American warning that the mission would be closed.

The State Department had declared the four unacceptable and ordered them to leave by last Monday, saying it had good evidence that the Libyans were participating in a campaign of intimidation against exiled opponents of Libyan leader Col. Muammar Quaddafi.

A State Department official said earlier this week that the department was concerned that the intimidation campaign could turn violent.

Speeches by Qaddafi and other official Libyan pronouncements since February repeatedly have called for "the physical liquidation of the enemies of the revolution abroad."

Earlier yesterday, the official Libyan news agency JANA quoted Qaddafi as saying he was "seriously thinking" of halting oil shipments to the United States and Britain, which also expressed concern about the activities of Libyan diplomats and reportedly is considering expelling several members of Libya's mission in London.

London was the scene of two of four recent murders of exiled Libyan dissidents.

The United States receives 10 percent of its oil imports from Libya, or 3.5 percent of its total supply. Britain receives less than 1 percent of its total supply from Tripoli.

Ali Houderi, the secretary of the Libyan mission, and Peter D. Constable, deputy assistant secretary for Near Eastern and South Asian affairs, disclosed the agreement in a brief joint statement at the State Department yesterday afternoon.

Houderi said Libya was "interested in having good relations with the United States" and that the four representatives had no desire to "contribute to a problem" between the two countries. He repeated his country's assertion, however, that the four men to be expelled, Nuri Ahmed Swedan, Ali Ramram, Mohammed Gamudi and Abdulla Zbedi, technically were not diplomat, but students.

Libyan students in the United States and seven European countries took over their embassies at Quaddafi's behest, restyling them "people's bureaus." The State Department continued to regard the mission here as a diplomatic establishment and said the Libyans had accepted this status.

One of the issues agreed upon yesterday was that the Libyans would negotiate to clarify the status of the members of the people's bureau and take steps to "conclude satisfactory arrangements" with the State Department.

Richard Shadyac, the American attorney for the Libyans, said the four men would probably leave the country Sunday evening. Until then, he said, they were allowed to move about the Washington area with few restrictions.

Soon after the settlement was announced, Houderi returned to the mission, where about 25 employes and Libyan supporters inside began dancing and chanting, accompanied by a drum, in what appeared to be a victory celebration.

At a press conference inside the mission, held in a large ground-floor room adorned with large portraits of Qaddafi and saying from the Libyan leader's "Green Book," the four reiterated earlier denials of any wrong-doing.

Zbedi, who informed sources said ran the cultural office that supervised a campaign to intimidate Libyan students in the United States, called the State Department charges "groundless" and said the department never gave specific evidence to support the charges.

He said it was "absolutely untrue" that Libyan students in the United States had been threatened in any way.

Meanwhile, outside the mission, about a dozen FBI agents and uniformed Secret Service officers maintained their informal vigil. The State Department had repeated earlier assurances that it had no plans to take the five-story building by force, since it was still considered diplomatic property and subject to immunity.

The State Department had issued several ultimatums to the Libyans to turn the four student-diplomats over or it "would have no choice but to require the closure of the Libyan" office.

A spokesman said authorities had no plans to take the building by force, but were prepared to cut off water, electricity and other utilities if no agreement had been reached by 4:30 p.m. yesterday.

The Libyans' lawyer, Shadyac, who said he was a "strong supporter" of President Carter and his new secretary of state, Edmund Muskie, said he deplored a comment President Carter made Thursday night at the swearing-in ceremony for Muskie in which he referred to the four diplomats as "the would-be assassins" hiding inside the Libyan mission.