M. Peter McPherson, administrator of the Agency for International Development, had an uneasy relationship with conservatives and tried last week to fix it.

It didn't work, and now they are trying to drive him from office in a campaign that also involves their opposition to several longstanding tenets of American foreign policy.

Conservatives have never been fans of foreign aid. They like it even less when it goes, as some now does, to Marxist governments. Of late they have also been stirred up over aid for family planning.

McPherson, hoping to come to terms with the conservatives, asked to speak last Thursday to The Stanton Group, a coalition of about 40 conservative organizations concerned with national defense and foreign policy issues.

It was one in a series of meetings he has held in recent weeks with legislators and special interest groups.

"It was a disaster," said Henry (Huck) Walther, executive vice president of the U.S. Defense Committee and co-chairman of the group. "Everyone left far more concerned about what Peter was doing than when he came in."

"I can't imagine why he wanted to come," said Paul Weyrich, the other co-chairman. "His answers weren't only out of synch with the group, they were weak and indecisive. It was an awful performance."

Weyrich, director of the Committee for Survival of a Free Congress, said he went into the meeting "basicially neutral" about McPherson. But by the end of the session he was vowing to organize a campaign to drive the AID administrator from office.

McPherson was in Japan yesterday, and not available for comment. Speaking for AID, Kate Semerod said he believes he has faithfully carried out the anti-abortion, pro-family-planning policies of the Reagan administration. According to participants, three issues gave McPherson the most trouble in his meeting:

* Relief assistance to famine victims in Ethiopia and Mozambique, both governed by Marxist regime. Conservatives charged that an administration decision to give about $37 million in drought assistance to Mozambique had propped up a government about to collapse. McPherson angered the group by saying he was only carrying out presidential policy, not creating it.

In Ethiopia, U.S. assistance is funneled through private agencies, but Walther said conservatives fear that the Marxist government there "is using starvation and our grain as a political tool."

* The use of empty military aircraft to transport food and medical relief to Central America as allowed under legislation approved by Congress last September. Andrew Messing, executive director of the National Defense Committee, is said to have charged that AID blocked tons of humanitarian aid from being airlifted on empty planes to combat zones in El Salvador.

* A $36 million AID grant to the U.N. Fund for Population Activities. AID had reducted its grant by $10 million because of concerns about compulsory abortions in China, which receives UNFPA funds.

Several members of Congress, including Sen. Jesse Helms (R-N.C.) and Rep. Jack Kemp (R-N.Y.), charged in a recent letter to President Reagan that AID ignored legislation approved by Congress last year banning support for the fund until it ends activities in countries that promote involuntary abortions.

McPherson reportedly told the group that the members of Congress didn't understand the legislation. "At that point," Weyrich said, "I turned to him and said, 'You're a disgrace. You are unfit to be in your current position. You do the president a great disservice. I want you to understand I favor your resignation and I'll do everything in my power to get this group here behind me toward that end.' " Weyrich said a show of hands indicated that everyone in the group of about 50 supported him.

A later meeting with Kemp went more smoothly, but Kemp press secretary John Buckley said the representative from New York "is certainly not happy about the way things are going at AID. He is not happy with McPherson."

Family-planning groups, which have had their own problems with McPherson, are somewhat baffled by the attacks on him from the right.

"I think he is getting a bum rap," said Joseph Speidel, vice president of the Population Crisis Committee. "He has carried out the administration's anti-abortion policy much to our dismay . . . . We've suffered a great deal of damage at the hands of McPherson. I don't see why conservatives are complaining."

Speidel added, "Peter McPherson believes in a foreign aid program. The far right not only wants to get us out of abortion around the globe, they're just against family planning."