Four major political pollsters said yesterday that the continuing controversy over budget director Bert Lance threatens serious damage to President Carter's standing with the public.

"Right now, it's just a little chink in the armor," said John Gorman, an executive of Cambridge Survey Research, the company that did Carter's campaign polling and that now works for the Democratic National Committee. "But if it keeps on like it's going, it's just a constant grinding negative."

Gorman's view that the Lance controversy jeopardizes Carter's eight-month-old honeymoon with the voters was endorsed, in separate interviews, by Democratic pollster Peter D. Hart, by Richard Wirthlin, who polled for Ronald Reagan in 1976, and by Robert Teeter, pollster for the 1972 Nixon and 1976 Ford campaigns.

But, on a busy day of White House denials that Lance may be forced to resign, Vice President Mondale rejected the suggestion that the Lance case is impairing the [WORD ILLEGIBLE] leadership position on other issues.

"I don't see that at [WORD ILLEGIBLE] Mondale said in an interview with The Washington Post. "In fairness to Lance, I think he deserves to [WORD ILLEGIBLE] it with on his merits, as anyone in his situation would want to be."

Mondale's comments came as the White House reported the President's mail running 2-to-1 against Lance, and press secretary Jody Powell rejected suggestions by two columnists friendly to the administration that a Lance resignation would be welcome.

Sen. James A. McClure (R-Idaho) joined those in Congress who have urged the director of the Office of Management and Budget to step down. "Lance was not straightforward with this country, and neither was the President," McClure said.

Carter has twice strongly defended Lance's controversial banking practices before he became budget director. Both Powell and Mondale in their comments yesterday tried to deflate speculation that the President was worried about the political cost of the continuing debate.

New York Times columnist James Reston said yesterday that "there has been a definite shift in the White House attitude" and Lance's fellow-Georgians now concede he "will probably have to go."

"That is not our feeling so far as I am able to speak for the senior staff," Powell said.

Powell also said he hoped Lance would ignore advice from Atlanta Constitution columnist Bill Shipp that he consider resigning to "save himself" from continuing criticism.

Shipp has been defending Lance in his column against what he described as a near-vendetta by elements of the Washington press and bureaucracy.

Lance, a longtime personal and political intimate of the President's, has been under investigation for his personal loans and for his management practices in two Georgia banks he headed. The comptroller of the currency cleared Lance two weeks ago of any criminal charges, but decided to continue his own investigation of other aspects of the case. Senate hearings into Lance's affairs are scheduled for next week.

Mondale was asked, during a discussion of foreign policy issues in Congress, whether the Lance controversy was damaging the administration's ability to press such measures as the Panama Canal treaty.

"I don't see it at all in my discussions with people," he said. Mondale also said he was unaware of any reconsideration of Lance's status in the administration.

But a very different appraisal of the actual and potential impact of the Lance case came from the four pollsters interviewed by The Post.

Gorman, chief partner in the firm headed by the President's pollster, Patrick Caddell (who was on vacation and unavailable, mat K5r this judgment:

"The longer it goes on, the more it is going to cut into the ratings on his (Carter's) ability to do his job. Not because he's regarded as dishonest himself, but because he looks indecisive and incapable of dealing with a crisis."

Hart, who polls for many Democratic senatorial and guernatorial candidates, said he found signs that Carter was "really dipping" in late-JUly and early-August polls in three scattered states, even before th Lance case became an issue.

He attributed that drop a return of the pessimism and cynicism that had dominated public attitudes for several years before the 1976 election.

In that context, he said, "the Lance thing is an absolute killer, because it strikes right at the heart of Carter's individual appeal. A candidate makes compact with the American voters, and his was to restore honesty to government and bring in competence. The handling of the Lance case undermines both."

"We always knew there would be a first crisis," Hart said, "but this is one of his (Carter's) own doing."

Wirthlin, who was Reagan's pollster and is president of Decision-Making Information in California, commented in a similar vein. "Every President makes his share of mistakes," he said. "What makes this so devastating to Carter is that it comes at his most vulnerable juncture, the issue of morality."

Wirthlin, who said he had seen the first effects of the case in polls this month in two Western states, said, "People don't care too much about the (Lance) overdrafts, but they do care about the double-standard Carter seems to be using.

"Carter's strength in the polls has been that he is a fresh breeze in Washington and a moral man.What's happened is that Bert Lance has started to open that up to question. I don't know how far it will go, but if the Carter administration feels that issue is behind them, they're sadly mistaken. Carter risked a great deal when he literally put his arm around Lance. The fact that he convinced people to vote their hopes rather than their fears makes it all the worse now."

Teeter, the vice president of Market Opinion Research Corp. in Detroit and chief pollster in the Nixon and Ford campaigns, said he had not data fresh enough to reflect the Lance case's possible impact.

But he commented that "what has amazed me is that people never learn from history. Early on, I thought Carter was handling it very well, but it sure sounds now like he's digging in, just like one of my former clients did. He begins to look like everybody else who's been there."

At the White House, Powell said "it is still our position that he (Lance) has done nothing that should warrant his being run out of the government, that we are willing to accept the difficulties that come with maintaining that position."

In San Antonio, where he was speaking to the Southern Governors United Press International as dismissing the reports of White House "wavering" as press speculation. "The strength and support of the White House has been very obvious," he said, "and I haven't given any thought to leaving."