Republican supporters of Ernest W. Lefever asked him to withdraw from consideration as the State Department's human rights chief because they feared his nomination would lose on the Senate floor after his Thursday closed-session testimony became public this week, according to well-placed Republican sources.

The senators -- Majority Leader Howard H. Baker Jr. (Tenn.), Jesse Helms (N.C.), Richard Lugar (Ind.) and S. I. Hayakawa (Calif.) -- were concerned that Lefever's performance Thursday, including his discussion of his relationship with the Swiss-based Nestle firm and his previous statements on Israel, had resulted in the negative vote of Sen. Nancy Landon Kassebaum (R-Kan.), according to Senate sources.

The senators had expected as many as four of the nine Republicans to join the eight Democrats in opposing Lefever, but they saw some possibility of winning in the full Senate, the sources said. Kassebaum was considered the bellwether of how well Lefever would do on the Senate floor.

The senators feared that publication of a transcript of the six-hour closed session would embarrass the White House by making its support for Lefever "look ridiculous," according to an administration source who was told later of the senators' concerns.

The committee had agreed to make the transcript public after Lefever had removed several references to his family and a Swiss foundation that provided seed money for the public interest center he still heads.

In his testimony at Thursday's executive session of the Foreign Relations Committee, Lefever contradicted his testimony before the same panel two weeks earlier. In his earlier testimony, he said a study of the controversy surrounding infant formula marketing in the Third World "was undertaken" by his Ethics and Public Policy Center and "the author chosen, without any contact, any financial contact, with Nestle," the world's largest manufacturer of infant formula and a contributor of $25,000 to the center.

On Thursday, according to two sources who were present, Lefever admitted under questioning by Helms, one of his staunchest committee supporters, that in September, 1979, a month before he first discussed the proposed study with its intended author, Fortune writer Herman Nickel, he both asked for a Nestle contribution to the center and told Nestle the center intended to conduct the study.

While admitting he discussed the contribution and the study in the same conversation, Lefever maintained that doing so did not violate the policy of the center not to accept contributions for specific projects in which a donor has a direct or indirect financial interest.

Republican senators, including Helms, Baker and Lugar, who represented three of the four votes for him in the committee, were stunned at Lefever's admissions, according to a Republican source.

Lefever "was not a very good witness in his own behalf," Baker said in a television interview with the Cable News Network yesterday. "To be frank with you we went into that thing (the Thursday session where Lefever was voted down by 13 to 4) with six votes, maybe seven, on the Republican side for Lefever and we came out with four."

Baker said that after the vote, "one senator and then another, and finally about six senators, contacted me, each separately and privately, to say they have great concern about this, and that they were . . . would find it more difficult to vote for him than they had responded initially."

But potentially even more troublesome to the Republicans was Lefever's response to questions by Sen. Paul Tsongas (D-Mass.) about the nominee's positions on Israel, the Republican source said.

Tsongas asked Lefever about a passage in Lefever's book, "Nuclear Arms in the Third World" in which he said "the United States should consider extending a nuclear guarantee to Egypt, Syria, and other Arab states" which would deter "the use of Israel's [nuclear] force for military purpose or blackmail."

Lefever said his words were being taken out of context and that he had urged nuclear assistance to both the Israelis and the Arabs.

When Tsongas read the paragraph aloud, Lefever said the entire chapter had to be read to show that he was talking about developing a system for nuclear stability in which Israel would be just as interested as the Arab states.

Lefever insisted the discussions of nuclear guarantees for Arab states constituted "academic speculations" that he had rejected, and he began to read long excerpts to support his point.

"Modest nuclear assistance to Israel compatible with the nonproliferation treaty is a small price to pay for helping to induce greater nuclear responsibility in a new nuclear state," Lefever read, according to one person present.

"Now wait a minute," Tsongas interrupted, asking if they were reading from the same book. Lefever had inserted the words "to Israel" into the text, Tsongas noted.

Lefever said he added the words so his real meaning would be clear to those who did not have a copy of the text before them.

Tsongas objected that the context nowhere implied the reference was meant to be applicable to Israel and in fact would indicate it was referring to the Arab states.

After several sharp exchanges with senators, Lefever objected that "no one can pin an anti-Israel label on me for anything I have said. . . . Any effort to pin a label on me at this point is without foundation," according to two persons at the hearing.

Tsongas asked him if anyone in the hearing room was trying to pin such a label on him. Lefever said no one was. He went on to say that in the mid-1950s he had criticized the United States for a U.N. vote condemning Israel for invading Egypt during the Suez crisis. He continues to hold pro-Israel views to the present, he said.