When Gregg Smith, a former Oregon housing administrator, left state government in 1983, he went into the housing business, developing retirement communities that he hoped would find a ready market in the fast-growing elderly population.

Like many housing developers, Smith turned to the federal government for support. The Capital Place Retirement Residence in Olympia, Wash., which opened in 1986, was built with state bonds and backed with Federal Housing Administration insurance available through a special program to encourage the construction of elderly housing.

Last year, however, when Smith tried to develop two new retirement projects, they fell victim to Housing and Urban Development Secretary Jack Kemp's efforts to clean up HUD. The Retirement Service Center program was one of the major targets of Kemp's reform efforts. Citing mismanagement that cost the government more than $100 million in bad loans, he canceled it.

Kemp also canceled or curtailed the Title X land development program, the Section 8 moderate rehabilitation program and the multifamily coinsurance system. Last month, the Senate adopted his plan to tighten eligibility standards for Federal Housing Administration mortgage insurance as part of its omnibus housing package.

In the view of many housing activists and developers, Kemp is acting on a not-so-hidden agenda. They charge that Kemp is using the climate of reform to promote his own conservative approach to housing policy, drastically reducing the role of the federal government and stressing private home ownership over federally-financed and -supported home construction.

"Each of these programs had some purpose, was performing some public policy mission," said Frank Shafroth, the director of federal relations for the National League of Cities. "The shutting down is occurring, but nothing is occurring to take its place."

Kemp's attempts to focus the department's efforts on programs more clearly targeted to low- and moderate-income people has begun to affect what has been a generally friendly relationship with Congress and local and state governments. His critics argue that Kemp's stated goals -- though noble-sounding -- are at odds with HUD's real purpose to serve all urban areas, middle income as well as poor.

The biggest program cancellations have involved insurance programs that commit the federal government to backing housing projects that the private market would not support. The policy initiatives he has proposed have involved a shift in emphasis rather than a massive commitment of funds and benefit relatively few people. They include Operation Bootstrap, designed to encourage local communities to provide job training, childcare and other assistance to public housing tenants, and a public housing tenant management program.

Smith, whose projects were never questioned by federal authorities, said, "I think they got so terrified of some of the spectacular failures that they said, 'Let's get out of this.' " He estimates that HUD's withdrawal from the elderly housing program has cost him "a couple of hundred thousand dollars."

Others affected by the cancellation of HUD programs share his view. A group of mortgage insurance companies joining forces as the Housing Study Group sued HUD earlier this year over its decision to discontinue the coinsurance program, a Reagan-era effort designed to encourage the private sector to share the burden of insuring low income apartment complexes. They won.

U.S. District Court Judge Joyce Hens Green ruled HUD tried to "trample upon the substantive rights and interest of the plaintiffs by interfering with and limiting" their ability to do business. HUD subsequently placed nine of 15 remaining coinsurers on probation. A Tennessee-based coinsurer, Benton Mortgage Co., has filed a similar challenge that is pending.

HUD officials suspended or killed the troubled programs as the first step in a series of efforts Kemp took to change the philosophical direction of an agency that had been created in the Johnson years as a massive shelter entitlement program. Along the way, Kemp admitted that the scandals were perhaps a "blessing in disguise" that allowed him to "clear the decks" of programs Republican administrations had never liked anyway.

For a time, Kemp's aggressive action was welcomed by the Democratic Congress as a solution to an embarrassing scandal that implicated oversight committees and Republican appointees as well as developers. Only recently have protests begun to emerge -- first among developers who feel they have been unfairly tainted by the scandal, and -- more slowly -- on Capitol Hill.

"The Reagan administration tried to shut it down through the means of the budget," said one former HUD official who still does business with the agency. "{Kemp}'s trying to shut it down by scaring the hell out of everybody."

Kemp and his aides vigorously deny that they are acting out of anything but concern over good management.

"That's bilge water," said HUD Under Secretary Alfred A. DelliBovi, referring to the criticism. "It comes from people who believe the Department of Housing and Urban Development should be the promotional arm of certain special interest groups and trade associations," said DelliBovi, who testified recently that potential losses from the Retirement Service Center program could have drained the FHA insurance fund by as much as $719 million.

Kemp points to the housing bill that recently passed the Senate, which includes many of his ideas and the department's fair housing enforcement efforts, as an example of HUD's intention to make housing available. He has also argued that federal money is better spent when targeted to low- and moderate-income households.

"I think we deserve some credit," he said. "Had there not been reform of the agency, then there would be no credibility on any of these subjects."

HUD General Counsel Francis A. Keating II describes the department's approach another way. "I think as a philosophical matter that smart caution is the appropriate {response} for operating a multibillion dollar enterprise," he said. "The reality is that under the preceding administration we ran the taxpayers' business right into the ground."

Barry Zigas of the National Low Income Housing Coalition argues that that situation won't change until HUD restores some of the monitoring staff that disappeared during the Reagan years.

"The question is, does the agency have the technical competence to review documents," said Zigas. "Is the agency really set up to take risks?"

DelliBovi, in testimony delivered before a Senate subcommittee last week, said HUD does not need to hire additional staff or ask for more money.

"The mere assertion that work is not completed on time or done correctly does not automatically translate into a need for more staff," he said.

Many in the housing industry also complain privately that Kemp, who likes to tout HUD as an economic development tool, has been overly critical of their industry since last year's scandals. "They have chosen this adversarial relationship . . . being on the side of tenants and against owners, which really is not realistic," said one developer.

Others who do business with HUD, most of whom asked to remain anonymous in order to preserve that relationship, linked Kemp's anti-poverty stratagems to political ambition.

"I think the secretary has made the mission of the department his election to president," said one attorney who has represented housing industry clients before HUD for 10 years.

Kemp said he is "offended" by such accusations. Such criticism comes, he said, "from an industry that is looking for a sweetheart deal from HUD that has been cut off."

Charles Edson, an attorney who represents the National Leased Housing Association expressed another common complaint. Kemp, he said, is turning HUD into a department with an exclusively anti-poverty mission. "I certainly strongly favor the anti-poverty activities," he said. "But this country also has to have an active housing agency."

"I don't think HUD ought to be in the middle class business," Kemp responded.

Sen. Bob Graham (D-Fla.), who has conducted his own series of hearings on what went wrong at HUD, said the decline in home ownership and housing starts points up the need for more, not less, federal input in housing.

Kemp's reforms, he said, were "sort of what you do in an emergency room when you have a patient who is hemorrhaging all over the operating table. You have to stabilize first." Once that stabilization is achieved, he said, HUD should undertake a broader social mission "that is bigger than bricks and mortar."

During debate over the National Affordable Housing Act -- the official name of the housing bill passed by the Senate -- Sen. Alan Cranston (D-Calif.), chairman of the Senate's housing subcommittee, wrote Kemp that the HUD scandals should not prevent the government from restoring some of the diminished national supply of low-cost housing.

"Many of those involved in the scandal were determined to destroy housing production programs," Cranston wrote. "We cannot let their corruption be used as an excuse to complete their hostile mission."