U.S. Court of Appeals Judge Malcolm R. Wilkeky asked the White House yesterday to withdraw his appointment to a lucrative membership on the Iran-United States Claims Tribunal -- an appointment that set off widespread grumbling among his colleagues.

President Reagan had announced Wilkey's appointment two weeks ago while the judge was out of the country. But State Department and White House sources said at the time that Wilkey had let it be known that he would be available for the position.

State Department sources said salaries for tribunal members, who will sit in the Hague or London to artibrate thousands of claims against Iran, would be "in the six figures." The job would be full-time and could last up to two years and perhaps longer, those sources said.

Court sources said the other 10 judges on the appeals court were upset because Wilkey's appointment would hamper their efforts to get the severely backlogged court up to date in its caseload. Wilkey apparently wanted to keep his seat while serving on the tribunal, thus depriving the court of one of its members for a long period of time.

The appointment also was a source of some embarrassement for the court, since it came a week before Chief Judge Carl CGowan asked the Senate for more judges to help cut the backlog. And Wilkey's colleagues also were troubled because the appointment seemed to violate a canon of the Judicial Code of Conduct, which flatly prohibits a judge "to act as an arbitrator or mediator," and warns federal judges not to accept such nonjudicial appointments if those would "interfere with the performance of his judicial

In the past, judges have participated on tribunals, notably Supreme Court Associate Justice Robert Jackson's service in Nuremberg after World War II and Chief Justice Earl Warren's tenure as had of the commission investigating the assassination of President John F. Kennedy. But both of those situations occurred under the old Code of Judicial Conduct, which was changed in 1973.

Wilkey's colleagues met in conference to discuss the situation soon after the nomination was announced, but decided to withhold judgment until Wilkey returned from a trip to Africa so they could discuss the matter with him. He returned a week ago, but the meeting with his colleagues never took place. By law, there is nothing the other judges could have done had Wilkey decided to accept the appointment. Only the Senate, by impeachment, could have forced him to choose one job or the other.

Earlier in the week the first rumblings of discontent within the court reached State Department and White House officials, who blamed each other for suggesting Wilkey's name and allowing the president to nominate him.

Reached at his home yesterday afternoon, Wilkey declined to comment. In his letter to the president yesterday Wilkey said that he asked to be taken off the tribunal after reevaluating the impact his absnce would have on the court's caseload.

The tribunal was established as part of the agreement that won freedom for the 52 American hostages in Iran. At this point, it appears there will be a total of ten members from the United States and another ten from Iran. Those twenty members will then select another ten members from then select another ten members from neutral third-party nations.