A political tug-of-war over a federal court of appeals judgeship in Utah has ended, after last-minute lobbying by House Speaker Thomas P. (Tip) O'Neill, with the selection of the brother of the state's only Democratic member of Congress.

In picking Monroe McKay, a Brigham Young University law school professor, earlier this week to fill the vacancy, President Carter bypassed the first choice of Attorney General Griffin B. Bell and the Utah state bar association.

The decision has triggered charges that in the Utah case the White House ignored Carter's campaign promise to keep politics out of the selection of judges.

The President said during his campaign last fall that "all federal judges and prosecutors should be appointed strictly on the basis of merit, without any consideration of political aspect or influence."

To help ensure that politics would not be a factor in high judicial appointments, Carter set up commissions to pick qualified nominees for courts of appeals vacancies.

There is no dispute about the qualifications of the two men involved in the Utah dispute. Both McKay, 49, and David K. Watkiss, 52, a Salt Lake City attorney, were among the five nominees forwarded to the White House by the 10th judicial circuit selection commission early last month.

But when Bell sent his personal recommendation to the White House a week or so later, he said his first choice was Watkiss, according to Michael J. Egan, the associate attorney general who handles such matters at Justice.

McKay was listed as Bell's next choice, Egan added.

By mid-August an FBI and American Bar Association check on Watkiss' background had been ordered, and the word spread quickly that Watkiss was the President's choice.

Egan said in an interview yesterday that the Justice Department had started the background check before the Prsident's decision because Congress was planning to adjourn in early October and it was hoped the nomination could be confirmed by the Senate before then.

"If this thing has caused a great flap, then the Justice Department obviously was precipitous in starting the FBI check when we did," Egan said.

When Congress returned from the August recess, Rep. Gunn McKay (D-Utah) began his own quiet campaign in support of his brother's nomination. He said, that he asked Speaker O'Neill early this month to "put in a good word for my brother with the President."

Rep. McKay said he asked O'Neill to make the approach "because I didn't think it would be kosher if I did it myself . . . But I don't feel a bit bashful about favoring my brother for the job."

Gary Hymel, a top O'Neill aide, confirmed that the Speaker had spoken to the President on McKay's behalf at each of the last two weekly leadership breakfasts at the White House. "Tip talked to the President twice because Gunn asked him twice," Hymel said.

Rep. McKay said that Frank Moore, the chief White House lobbyst, called him Tuesday to say that the President had decided to appoint his brother.

White House aides later confirmed that Monroe McKay was the President's choice, pending only the outcome of FBI and ABA checks on him that Justice was directed to start on Tuesday.

The White House aides declined to say what impact O'Neills intervention had on the President's decision. They pointed out only that both men were qualified for the job, and that the President, not the Attorney General or the state bar association, makes the decision.

Backers of Watkiss were critical of the White House shift to McKay, but acknowledged they had lobbied for their candidate too.

Utah Gov. Scott M. Matheson, a Democrate, called White House aide Jack Watson early this month to announce his strong support for Watkiss, his campaign chairman last fall.

The governor also asked the state bar association to rate the candidates and sent the results to the White House, according to James B. Lee, the bar association president. Lee said Watkiss ranked first, McKay third of the five candidates.

Mike Graham, Matheson's administrative assistant, said of the episode, "What can you say except that this shows who has more stroke, Tip O'Neill."

Egan said the Justice Departmnet had no complaints about the decision, but another high Justice official said the department was concerned about the congressional lobbying in the case.

Asked about the Utah case in light of Carter's pronouncement about the nonpolitical selection of judges, Egan said, "The idea of the selection commissions was to pick good people so we wouldn't be pressured to put someone on there [the federal bench] who's not qualified."

And pressure for qualified people? "That's the name of the political game," Egan said. The President's statement about divorcing politics from judicial selection "just isn't realistic," he added.