The Bert Lance affair is beginning to do serious damage to the Carter administration, a varied group of political figures said over the weekend.

At least two joined the chorus of calls for the budget director's resignation.

Others - citing what they called the "serious nature" of the charges swirling about Lance's tangled financial dealings - said they would withhold judgment until the embattled Georgia banker lays his case before the Senate Governmental Affairs Committee on Thursday.

"I think the President has been very badly hurt," said Delaware Gov. Pierre S. du Pont, a Republican. "The issue is not Mr. Lance, who I believe ought to resign and I think will resign. The issue is the credibility of the President . . ."

Du Pont was one of five governors appearing on "Meet the Press" (NBC. WRC), broadcast at the end of the annual meeting of the National Governors' Conference in Detroit. All five said the Lance affair has hurt Carter: only du Pont called for his resignation.

Lance's problems stem from a series of alleged irregularities at his Calhoun. Ga., First National Bank and at National Band of Georgia in Atlanta, of which he became president in 1975. The complicated financial transactions include overdrafts of up to $400,000 in checking account by Lance and his family at banks they control.

Du Pont said yesterday he doesn't think the public "clearly understands six-figure overdrafts and double pledging of collateral and so forth.

"But the public does understand that Jimmy Carter said to us last fall that we're going to have a better ethical standard in government. We have seen the President muff that hot grounder."

Carter's own assistant for public liaison, MudgeCostanza, said the controvery has proved a "burden" for the administration and suggested that Lance step down.

"I think that in light of the burden that this issue has brought upon the President. Bert Lance is prevented from doing the job he was hired to do . . . And I think Bert Lance should relieve the President of this burden," she said.

Asked if she meant Lance should resign, she replied: "Of course," Costanna was interviewed for radio in Rochester, N.Y.

"I'm not asking him to resign at this point," said Sen. Henry M. Jackson (D-Wash.) a member of the committee that will hear Lance this week. "I've not heard Mr. Lance, and I think that decency requires that he be heard."

However, Jackson said he believes that disclosures about Lance's financial activities, and his responses to those disclosures, have already placed him" in a very difficult position."

They have also hurt the President, Jackson said. But he added: "I've been around long enough to know that it's rather foolish to make a contemporaneous judgement . . . This temporary decline in his (Carter's) popularity . . . may change overnight, depending on future events . . . But this has had an impact . . ." Jackson appeared on Face the Nation (CBS, WTOP).

Jackson said one of "the major items of inquiry" at this week's hearing will be whether the administration withheld damaging information about Lance during his Senate confirmation hearings.

"There have been all sorts of allegations about information that was available to the White House staff but was not forwarded" to the Senate. Jackson said. "This is a crucial part of the whole investigation because the nomination process and confirmation process cannot be truly effective unless we have all of the information. That goes to the credibility of all future nominees."

Senate Republican leader Howard H. Baker (Temn.) said yesterday that Carter "has been hurt" politically by Lance's problems, and he predicted in the 1978 elections, that the issue will help Republicans.

"I really feel sorry for Bert lance. I do," Baker said in another NBC television interview. "But you know the transgressions that are charged and alleged against Lance - it that was a Republican director of the OMB (Office of Management and Budget), they would have hung him from a lamppost right now."

House Republican Leader John J. Rhodes (Ariz), interviewed on the same program, said he shares Baker's sentiments. The Lance controversy, coupled with other actions by the Carted administration, has given Republicans more 1973 election issues "than you can really say grace over," Rhodes said.

He predicted that the growing push for Lance's resignation should "come to a head very shortly."

Rhodes said Republicans until recently have been reluctant to target the Lance issue because "this is more or less the President's business."

He came into . . . office with campaign promises about how pure his administration was going to be and I think it is up to him to decide whether or not it [the Lance case] meet with that purity," Rhodes said.

About the only voice publicly supporting Lance over the weekend was that of United Nations Ambassador Andrew Young, who is also a good friend of the President.

Young said Lance has been unfairly treated by the press. "We've heard the prosecution, but we haven't heard the defense," he said. "A person is innocent until proven guilty. That's not something that can be done in the press."