The Reagan administration has failed to persuade Western European countries of the need to isolate Libya because of its alleged support for terrorism but has secured pledges that they will not undercut U.S. sanctions.

Deputy Secretary of State John C. Whitehead, who has been visiting Western European capitals in an attempt to drum up support for moves to isolate Libya, told a press conference here today that France had become the latest country to agree not to take advantage of a U.S. trade embargo of Libya. But he conceded that he had not achieved "as much as I would have liked" during a trip that has taken him to Canada, Britain, Italy, West Germany, and Greece.

Whitehead said that he had sought to persuade the governments to cut oil imports from Libya, reduce air traffic with Tripoli, tighten travel controls on Libyans, and cut staffs of Libyan embassies.

The list appeared to mark a significant scaling back of U.S. expectations since Jan. 7 when President Reagan called on Western European countries "to join with us in isolating" Libyan leader Muammar Qaddafi. Many of the governments have expressed reluctance to impose extensive sanctions.

Insisting that he had supplied U.S. allies with irrefutable evidence of Libyan involvement in terrorism, Whitehead said it was "hard to understand" the European resistance to sanctions.

"I think people are cautious about taking drastic action against a sovereign government so there is hesitancy but I can't explain why it has been so difficult," he said.

French Foreign Minister Roland Dumas today justified France's reluctance to consider further sanctions against Libya by saying that it had already cut back its trade with Tripoli to a quarter of the 1981 level. He said that the U.S. sanctions corresponded to the much higher level of U.S. involvement in the Libyan economy.

Western European countries are to consider what action to take against Libya at a meeting of Common Market foreign ministers in Brussels next week.

A strong indication that European countries will turn down proposals for an economic boycott of Libya came today from Dutch Foreign Minister Hans van den Broek, acting chairman of the European Council. He told a news conference in the Hague that it would be very difficult to reach a consensus on "the economic isolation of one country."

He indicated, however, that European Community countries might be ready to send a "signal" by formally condemning states that harbor terrorists. He also stressed that the community would "in no way" undercut U.S. economic sanctions against Libya.

The European Parliament last week rejected antiterrorism resolutions mentioning Libya by name, as well as economic sanctions. It adopted a blanket resolution condemning international terrorism.