The plot was the same, only the name of the leading character had changed yesterday as Republicans on the Senate Rules Committee spent three hours grilling one of President Carter's nominees to the Federal Election Commission.

This time it was John McGarry on the witness stand, a special counsel to the House Administration Committee and friend of House Speaker Thomas P. (Tip) O'Neill Jr., whom Carter has designated for a Democratic seat on the FEC.

Republican protest last week over the nomination of Sam Zagoria, director of labor-management relations for the U.S. Conference of Mayors, lead to Zagoria's request Saturday that Carter withdraw his name for a Republican seat. Carter agreed to do so.

Yesterday, Sen. Mark O. Hatfield (R-Ore.) and Sen. Robert P. Griffin (R-Mich. sharply questioned McGarry about his personal tax records and about financial disclosure statement McGarry filed with the House.

Exasperated at being unable to get the answers he wanted, Griffin finally told McGarry that his financial statements "certainly raise a question about what kind of job you would do as a FEC commissioners."

McGarry repeatedly told the senators that the answers to financial questions were not clear-cut, involving "an interplay of accounting principles, tax principles and legal principles," and that the Internal Revenue Service had approved his figures.

Hatfield and Griffin focused on a $70,000 payment McGarry received from his former law partner, Irving H. Sheff, in 1976, and two payments of about $10,000 and McGarry received in 1975 and 1976 from a Massachusetts vending company.

The $70,000 was paid to settle a contract Sheff and McGarry signed in 1973 when their law partnership was dissolved.

Hatfield, reading from the 1973 agreement, said the payment represented income for legal services and should have been included on the House disclosure form and as income subject to self-employment taxes.

McGarry said certain language in the contract was "just boilerplate" and disagreed with Hatfield's conclusions.

The $10,000 payments from Melo Tone Vending Inc. were for legal services McGarry had performed several years earlier. The 1975 he reported the money as income from his legal work and included it in the money subject to self-employment taxes. But in 1976 he reported the money to the IRS simploy as miscellaneous other income, not subject to the self-employment levy.

McGarry did not explain why he handled the sums differently.

Sen. Claiborne Pell (D-R.I.) questioned McGarry about a $4,950 investment in a real estate tax shelter that resulted in about $176,000 in tax deductions.

"I'm not indicating there was anything illegal," Pell said, asking McGarry to explain the transaction. "I'm just curious for myself . . . I want to congratulate your accountants on their ingenuity."

McGarry said he believed the $4,950 figure was too low, but did not explain the transaction, saying it had been devised by his accountants and tax lawyers.

Griffin read into the record a letter from Common Cause opposing McGarry's nomination because of his ties to members of Congress. A copy of the letter had been sent to each committee member.

"Mr. McGarry is being asked to oversee the campaign finance activities of members of the House for whom he worked for a number of years and who are directly responsible for his obtaining this appointment," Common Cause said.

"It is of particular importance that those chosen to lead the commission be public servants capable of building and maintaining the credibility of the FEC. This is simply not the case with Mr. McGarry."

In agreeing to withdraw San Zagoria's name from consideration for the FEC, the White House elicited a promise from Republicans not to unnecessarily prolong debate over McGarry. But the Republicans indicated they had certain questions they would insist be answered.

Yesterday, Pell pushed for an afternoon session in hopes of reaching a committee vote on the nomination today. But Hatfield and Griffin urged a recess instead.