Rumors of Jimmy Carter's budget cuts have been flying like inflation, making the National Urban Coalition people good and worried. Good and mad, too.

"It's been a series of empty promises and broken dreams," was how one coalition supporter summed up Carter's approach.

But by yesterday afternoon, the president had soothed fears by having the coalition leadership over to the White House. And by the time of the coalition's annual "Salute to the Cities" dinner last night, some members were allowing that, well, it is awfully hard to be president these days.

"Listening to him this afternoon, I felt much better," said Richard Hatcher, the Gary, Ind., mayor and coalition steering committee member. "I have a sense that the critical programs the cities absolutely need -- like jobs programs, housing programs -- are going to be left pretty much intact."

"He's a very decent man," said William Ross, the executive director of a national job training program and another steering committee member. "I don't feel that he wants major cuts in social programs. But we've got to watch the Congress. They don't want to hear nothing."

The budget talk meandering through the coalition dinner at the Shoreham Americana was in fact all speculative, given that Carter had said he won't announce specifics in his estimated $15.5 billion worth of 1981 budget cuts until Monday. Still, Secretary of Labor Ray Marshall put in an appearance at a before-dinner reception to assure anyone who asked -- and lots did -- that CETA as well as federal youth programs would survive. "There's a lot of concern," Marshall admitted.

But Marshall's appearance and Carter's 45-minute White House talk weren't enough for everyone.

"You cannot make symbolic cuts which will inflict real pain and suffering on people," said coalition president M. Carl Holman. "I think that Carter's record in terms of the cities and the poor is a mixed record."

Then, echoing a sentiment heard from other coalition leaders during the dinner, Holman took on Capitol Hill.

"Blacks elected a hell of a lot of congressmen and senators who are now turning their backs as far as the budget is concerned," he said. Earlier this month, the House Budget Committee recommended $16.4 billion in budget cuts, including $1 billion in aid to the cities.

The coalition dinner chairman was W. Michael Blumenthal, the Treasury secretary that Carter fired in the great Cabinet purge last summer. Blumenthal is about to become chairman of the board of Burroughs Corp. Last night, he expressed near glee at his return to the private sector.

"I'm very happy to be the master of my own destiny," he said. "The times are very difficult. The economy is in worse shape than anybody expected."

About 900 people showed up for the dinner, which featured stuffed squab and a speech from Mayor Marion Barry, who took a Ted Kennedyism and revised it for his own purposes.

"It bothers me that when a baby cries in Anacostia the mother has to worry if it is a $50 cry or a $300 cry," Barry said. This was an example in the speech that decried Carter's budget cuts. The mayor has also been forced to cut his own budget, but earlier he expressed no kinship with the president he's on record as supporting.

"No, I don't have any sympathies with him," Barry said, asserting that too much of the taxpayers' money is being spent on defense and not enough on domestic programs.

At the dinner, the coalition honored 14 persons for their contributions to American cities. Among them were special Middle East envoy Sol Linowitz, who was in the Middle East, Time Inc. board chairman Andrew Heiskell and Hatcher, who was given the coalition's Distinguished Urban Mayor Award.