President Carter, turning his fire on Ronald Reagan and other Republican critics, said today that calls for a large tax cut and rollbacks in government programs to end the recession are "political double talk and ideological nonsense."

At the same time, addressing the annual meeting of the U.S. Conference of Mayors, the president strongly defended his record of providing aid to the cities and said that if the recession worsens seriously he will take additional, unspecified steps to cushion its effects. But under no circumstances, Carter said, will he "refuel inflation" or back down from his determination to impose "fiscal discipline" on the federal government.

"I'm determined to restore economic leadership without turning our backs on the poor or the elderly or the afflicted or the deprived or those who have suffered too long from the effects of racial or other discrimination," he said. "I reject the easy promise that massive tax cuts and arbitrary rollbacks of government programs are the answer. Such facile, quick fixes should be recognized as political double talk and ideological nonsense."

The jab at Reagan, whom he never mentioned by name, elicited tepid applause from the largely Democratic audience -- the only time the president's 34-minute speech was interrupted. Although Carter, in a long recitation of administration accomplishments, asserted that he had "stopped federal neglect of our cities," his message that he will resist domestic spending increases was not what the mayors wanted to hear.

Many mayors, especially those from steel and auto cities hardest hit by the recession, expected Carter to "bring something" for them to the conference. Throughout the meeting here, the mayors have been working on a set of anti-recession proposals, including expansion of federal job-training programs, extension of unemployment benefits for at least another 13 weeks, and additional funds to cities faced with declining revenues because of high unemployment.

"An empty bag," a big-city mayor said of the president's speech. Mocking Carter's references to the need for "fiscal discipline," this mayor added:

"Yeah, fiscal discipline and be unelected. He might come off that as time moves on."

But many of the mayors saw a glimmer of hope in the president's promise to act if the recession deepens. Mayor Coleman Young of Detroit, a strong Carter supporter, said, "Until we all begin to speak the same language, until the recession hits the whole nation, I don't believe we'll get some action. I believe it's just a matter of weeks before the message is full-blown."

Overall, the mayors, who have generally been among Carter's strongest political supporters, appeared unimpressed with his "balanced budget," which they view as a heavy burden on their communities.

Carter's appearance here was officially a "nonpolitical" event. But the political overtones were unmistakable as the president defended his economic policies to the chief executives of the nation's traditionaly Democratic big cities that are key to his chances for reelection.

The politics of the day were also compounded by the flap here over charges that the mayor's conference buckled under White House pressure and withdrew an invitation to Carter's Democratic rival, Sen. Edward M. Kennedy (Mass.), to address the group today.

According to White House press secretary Jody Powell, Kennedy had agreed to speak here Wednesday but, learning of the president's appearance today, later insisted that his speech be scheduled for the same day. Powell conceded that White House displeasure with the idea was conveyed to conference officials by presidential assistant Anne Wexler.

"Our position was that they should stick with the schedule they gave us," he said.

Wexler issued an implied threat that Carter would cancel his speech if Kennedy showed up on the same day. In the process, said an official who is sympathetic to the administration, White House officials badly blundered by providing Kennedy with more publicity than he would have received by speaking here.

"The White House shot itself in the other foot," the official said.

Kennedy sympathizers, meanwhile, made sure that copies of the senator's speech today to the American Federation of State, County and Municipal Employes in Anaheim, Calif., were distributed to the mayors here.

Later, Democratic mayors not only endorsed Carter but also approved a resolution urging Kennedy to withdraw.

"We would consider his continuation as a candidate obstructive and not in the best interests of the Democratic party," said Mayor Lee Alexander of Syracuse, head of the conference Democratic caucus. "I think he's caught up in the combat of the situation. He's still running and I don't think he realizes that the battle is clearly over for him."

Powell played down the significance of the incident Monday night when angry blacks booed the president and threw bottles at his motorcade in Liberty City, the section of Miami where rioting erupted last month.

"He's glad he went there," Powell said. "That sort of thing can always happen when a president travels. We felt that if he did not go into Liberty City the visit would not mean what we wanted it to mean to the people in the area . . . He considers that part of the job."

Many big-city mayors here are deeply troubled by the Miami rioting, fearing that, with unemployment rising, violence could also strike their cities. Administration officials have sought to calm these fears, fanning out over the conference to tell some mayors privately that there may be additional money for summer jobs for youths.

A White House official said it is not yet clear how much additional funds would be allocated. "What we're doing is scouring the budget for every last dollar that can be used to create jobs," he said.

But there was no mention of additional money today by the president. He defended his decision to increase defense spending while acknowledging that it was highly unpopular with the mayors who are looking for more aid to their cities.

"The decisions that I've described to you on defense, energy, self-discipline in Washington, will be carried out," Carter said. "You need to understand them. But this need not provide any obstacle between us in working for the alleviation of suffering, the caring for the poor, the rebuilding of our cities."

En route back to Washington, Carter stopped for two hours in Grand Island, Neb., to view the destruction left by a battery of tornadoes last week.

Talking to disaster victims, the president promised federal aid and said the government was eager to help.

The tornadoes killed five people and injured more than 200. Property damage was estimated at $140 million.