The Department of Housing and Urban Development reacted angrily this week to congressional criticism that the agency's regulations for distributing federal funds to religious organizations operating homeless shelters are too restrictive.

The rules "go to the outer limit we can" in channeling money to religious groups without violating the constitutional requirement for separation of church and state, said Robert S. Kenison, HUD's associate general counsel for assisted housing and community development. He said the agency agrees with its congressional critics that churches play "a vital role" in helping homeless Americans.

Congress has appropriated $370 million for assistance to homeless Americans during 1987 and 1988, much of it for housing. HUD regulations say that religious organizations that want federal funds must first establish separate companies or other legal entities to which they can lease the property to be used for shelters.

A recent report by the House Government Operations Committee's housing subcommittee said this solution "is neither constitutionally required nor very practical." Barriers should not be erected against distributing money to churches and other religious organizations, which provide many of the shelters and other aid for the nation's homeless population, the report said.

HUD changed its regulations in October in response to private and congressional criticism of rules it proposed earlier this year. Those rules said that before applying for federal money, religious organizations would have to sell property they wanted to use as shelters. Critics said such sales would be costly, difficult and sometimes impossible for religious organizations.

The change to allow for leasing instead of selling the property is "significant, dramatic and, I hope, successful," Kenison said, adding that no Supreme Court decisions exist to guide the agency in dealing with the question.

J. Michael Dorsey, HUD's general counsel, said: "We did not intend to stonewall or to make life difficult for the agencies that would be administering this program. We have an obligation when we issue regulations to make sure they do not violate the constitution." The requirement for sale of property to a separate entity has been used in regulations for Community Development Block Grants and other housing and development programs, he said.

The HUD officials questioned estimates by advocates for the homeless that 2 million to 3 million Americans are homeless, saying they have not seen any studies or surveys supporting these figures. In 1983, the department estimated the number of homeless at 250,000 to 350,000, but now believes there could be as many as 650,000, according to James W. Stimpson, deputy assistant secretary for policy development. A private study by the National Bureau of Economic Research estimated in the fall of 1986 that 350,000 people are without homes on any one night, he said.

HUD is trying to derive a reliable estimate of the number of people who are homeless on any given night so the agency can determine the extent of the need for shelters, Stimpson said.

Maria Foscarinis, of the National Coalition for the Homeless, said the estimate of 2 million to 3 million homeless Americans is based on reports from "the people on the front lines," operators of shelters and other facilities for the homeless. "It is impossible to know exactly" how many people are without homes, she said, adding that "one trend we are seeing is that homelessness is spreading beyond the cities to suburban and smaller areas."

The HUD officials said criticism that distribution of funds for shelter operators had been delayed unnecessarily was unfounded. They said a deadline was missed in only one of five HUD programs established last spring by the Stewart B. McKinney Homeless Assistance Act, and funds for the others were distributed more rapidly than is usual with federal programs.