The Office of Management and Budget is seeking to kill the government's largest aid program for cities and drastically cut back the third-largest and now fastest-growing federal welfare program, subsidized housing for the poor, sources in and out of government said yesterday.

The housing program might be supplanted in part by a new and smaller system of housing vouchers, sources said.

The contemplated cuts are certain further to aggrieve city and state officials upset by the cuts in aid the Reagan administration already has made.

To placate and to compensate cities for at least part of their loss, Richard Williamson, head of White House liaison with cities and states, said in an interview during a meeting of the National League of Cities in Detroit that the administration probably would not seek to phase out the $4.6 billion general revenue sharing program for local governments as once threatened, and in fact might move to increase it somewhat.

The possible urban and housing budget cuts are the latest in a series surfacing in recent days as the administration has refined its forthcoming budget for fiscal 1983. Administration officials have made it clear that this will involve further large spending cuts, mostly in domestic programs, in an effort to hold down a 1983 deficit that many experts now think will exceed $100 billion.

The political problems this may cause the administration became clear at the League of Cities meeting. The predictable criticism came from Democrats. New York Mayor Edward I. Koch attacked "the sham and shame of the new federalism," which he said "imperils the cities and is wholly lacking in realism and responsibility."

But Republicans were also critical. Guest speaker Mark O. Hatfield (R-Ore.), chairman of the Senate Appropriations Committee, criticized the administration for cutting domestic programs while increasing defense. And the most stinging criticism of the day came from Vermont Gov. Richard Snelling, a Republican and chairman of the National Governors Association.

"Frankly, I think that what is happening is that we are having an economic Bay of Pigs," Snelling said. "What is happening is that . . . we get tax cuts, which pleases the supply siders. We get budget cuts, which pleases the monetarists. We get dribs and drabs of federalism and we talk about it a lot, which appeases the federalists, and we sure are whacking away at the substance of government . . . . The problem with this potpourri is that it is in fact not an economic policy. Any of the above would be an economic policy. What we have is not an economic policy."

Snelling appealed to the assembled officials to join the governors in seeking an economic summit with the president to discuss budget cuts.

While Snelling was speaking in Detroit, the U.S. Conference of Mayors reported here that OMB Director David A. Stockman had notified the Department of Housing and Urban Development that he wants to phase out by 1984 the existing program of community development block grants and a companion program of urban development action grants. He has also told the department he wants to provide authority in fiscal 1983 for no new subsidized housing units for the poor.

In recent years, the number of new units authorized annually for these programs for the poor, which include the so-called Section 8 program and public housing, has been running at more than 200,000. In fiscal 1982, the Republicans cut this, but only to about 150,000. The housing programs are expected to cost nearly $10 billion in fiscal 1983, behind only Medicaid and food stamps for the poor.

Neither HUD nor OMB would comment on the contemplated cuts. The Stockman proposal is not final; HUD Secretary Samuel R. Pierce Jr., said to be angry over the proposal, has the right to appeal to the president, who plans to hold a series of budget meetings with Cabinet officers starting this week.

Sources said Stockman wants to reduce the community and urban development grant programs from the $3.6 billion authorized in fiscal 1982 to $2.4 billion in 1982, $1.2 billion in 1983 and eliminate new budget authority altogether starting in fiscal 1984.

Sources said Stockman wants to cancel some of the 150,000 new subsidized housing units Congress voted for this year, then eliminate all new authorizations for the low-income units in fiscal 1983, reversing two generations of federal housing policy. Other sources said a handful of new units might be funded in fiscal 1983. HUD's own request for fiscal 1983 was for an added 100,000 to 144,000 units.

In place of the existing programs, Stockman reportedly is considering giving large numbers of low-income households, perhaps as many as 200,000, vouchers worth small amounts of money which they could apply against their rents. This idea has been put forward by the president's advisory committee on housing as less costly and otherwise preferable to existing subsidies, which commit the government to keep up payments over periods of 15 years or more.