About 250 mayors sat silently yesterday as Sen. Edward M. Kennedy (D-Mass.) told them that all levels of government must find "more effective ways to cut the fraud and the fat and waste" in public programs.

Kennedy, who was warmly applauded at many points in his address to the U.S. Conference of Mayors here, got no response when he said the cuts must be made "in accord with the rising costs and a rising frustration of the taxpayers who pay the bills."

Many mayors here have expressed reluctance to make further budget cuts and have called for more state and federal spending to pay for health, education and welfare services now financed by the beleaguered local property tax.

Kennedy told them that "none of us will find it easy to perform the surgery required. But the operation must be carried out if we are to restore the confidence of the people . . . If we are not willing to cut the fact, then others who care less abut those urgent problems may rush in to cut the muscle, too."

The senator joked about California's June 6 vote for Proposition 13, a constitutional amendment that slashes local property taxes. "A month ago, people in Washington thought Proposition 13 was a new rock group on the West Coast. Well, they know a lot different now," he said.

Kennedy said the Proposition 13 vote should not be used to destroy local tax systems or make mayors more dependent on states and the federal government. "Above all, if we are going to balance the federal budget, let us resolve that the burden will be shared by all - not just the cities, not just the poor and the black and the sick," he said.

For instance, he said that if welfare programs are cut to stop wasted revenues, tax loopholes that help the rich should also be cut. "If we are going to provide tax relief from the burdens of soaring costs, let us do so in ways that do not undermine the cities' public schools," he said. Kennedy has strongly opposed a House-passed bill to provide tax credits for private college and secondary school tuition.

He said it is unfair to blame inflation and unemployment on the cities or their mayors. "In large part the blame falls squarely on the shortsighted policies of the federal government. Now is the time that our federal policies have to change and start paying more attention to the cities," he said.

The senator, who was invited to the convention after President Carter turned down an invitation, praised the administration for announcing a new urban program last March.

He was roundly applauded when he said, "The time has come for the federal government to give greater breathing room to the mayors and let them do their jobs. If we can deregulate the airlines . . . we can also deregulate the mayors so that they can serve the people of our cities." He called for cutting red tape in procession urban applications for federal funds.

Earlier, in a welcoming address, Atlanta Mayor Maynard Jackson blamed most state legislatures for not giving "home rule on taxes to America's mayors" and for "imposing on cities and towns, costly requirements" to carry out social programs "without providing consistent resources."

He suggested that the conference meet with the National Governors' Association about tax reform and "make it crystal clear: we, too, are mad as hell adn we're not going to take it any more either."

Sam Brown, director of the federal volunteer program Action, said he thinks "what is now called the 'taxpayer revolt' is not necessarily a national phenomenon." California's situation - with a 35 percent state budget surplus, 25 percent-a-year increase in property values and a recent jump in real estate taxes as high as 300 percent - makes it atypical, he said.

"What I believe we are hearing is that government is distant and not responsive to the best interests of our citizens," he added.

If officials interpret the California vote as a call "to make across-the-board cuts in programs that help the poor, the aged, the young and the ill . . ., we run the risk of using the taxpayers' revolt to reinforce social barriers and class distinction," Brown warned.

Brown said Carter had responded to citizen unrest by including in his urban package "programs based on microbucks" - several neighborhood self-help and volunteer programs designed "to reestablish a theme lost in America - the notion of neighborhood and community responsibility."