President Carter's chief domestic adviser yesterday strenously denied accusations that the administration has backed away from its commitment to urban America.

The defense came as, separately, the Congressional Black Caucus charged that Carter's State of the Union message was "totally irrelevent" to the needs of the poor and that his budget proposal is "immoral, unjust and inequitable."

Caucus members vowed an all-out fight against the budget and called for a meeting of national black political leaders to plan ways to "reverse spending priorities in the budget."

Their complaints echoed those of other black leaders and organized labor and came a day after John J. Gunther, executive director of the U.S. Conference of Mayors, charged that the Carter budget for 1980 "indicates an abandonment of earlier policies and program commitments."

Stuart E. Eizenstat, Carter's top domestic adviser, said in an emotional speech to mayors attending the mayors' 46th annual midwinter conference here, "There is no way that we are backing away" from the President's urban commitment.

Administration officials worked a year on the urban program, Eizenstat said, adding that they have no intention of letting it "go down the drain." In fact, he said, "in difficult economic times, we are trying as fast as possible to implement it."

In a strong attack Wednesday Gunther revealed a conference staff budget analysis which attacked Carter's drive to pare federal spending as a share of the nation's economic output, questioned whether cutting the federal deficit would reduce inflation and charged that Carter was giving up on some of his urban programs.

Yesterday several mayors disputed those conclusions and a subcommittee was created to review them and recommend by today whether the conference should reject them. Such a rebuke to Gunther, who has headed the staff since 1960, would be unprecedented. Several mayors said the conference may find "compromise language" to resolve the different viewpoints.

Mayor David Rusk of Albuquerque moved to reject the staff conclusions. He called many of the judgments "not well founded, exaggerated and not balanced."

Later, in an interview, Denver Mayor William H. McNichols, this year's conference president, said, "Carter has made an honest effort to be fair. He did what he had to do."

Detroit Mayor Coleman A. Young, who heads the conference's urban economics committee, said he thought a consensus of mayors would not support the staff conclusions. "I don't think it's evident that Carter is going back on his urban commitment," Young said. "I don't think it's a fair assessment."

Mayor Donald J. Canney of Cedar Rapids, Iowa, said, "I'm not opposed to the budget at all. We've got to knock down the budget deficit. Sure, programs that help people in cities will be curtailed and the needy will be hurt.But we have to cut. The funds are just not available."

Eizenstat, addressing the urban economics committee, said Carter had not resubmitted certain urban programs that died in Congress last year because "Congress gave us a clear signal it was not going to pass them."

He said that of the 13 parts of the urban package that were approved last year "all will be fully funded." He said no funds were sought for a private jobs program in 1980 because it received $400 million in supplemental funds in the current year and additional funds "are not necessary."

Elzenstat also noted that Carter is seeking a scaled-down version of his proposal for supplemental aid to cities and has proposed creation of a national development bank, both of which failed last year. "We haven't done all that we would have liked," he said. "We believe we've done all that we possibly could do."

At a news conference, Black Caucus Chairwoman Cardiss Collins (D-Ill.), speaking on behalf of the group's 17 House members, said the caucus "cannot and will not accept a policy which is a continuation of the discredited trade-off theory that uses unemployment as a weapon in the attack on inflation."

Caucus sources said later they hoped a meeting of national black leaders could be called in the next 10 days.

Echoing comments a week ago by National Urban League President Vernon E. Jordan, Rep. Parren Mitchell (D-Md.), former caucus chairman, warned that blacks are approaching a "toleration level for frustration."

If that level is broken, he said, "I hope no one calls me to go out and try to 'cool it.' I will not play that role. I think for me to play that role would be to betray the interests of the black community."