Daniel W. Horgan, who quit his White House consulting job this week after only four days, is the latest casualty in a growing New Jersey scandal that has already shaken the state's housing program and sparked three law-enforcement investigations.

Horgan, 48, had been executive director of the Democratic Nationa Committee before going to the White House. He ran President Carter's 1976 campaign in Ohio, and had been expected to play a key role in his reelection campaign in the Northeast.

But he abruptly quit his White House consultant's post Thursday after The Trenton Times linked him to an alleged payoff deal involving the state's Housing Finance Agency (HFA).

Before being tapped to run Carter's Ohio campaign, Horgan had been the second-ranking official in the New Jersey department that oversees the state HFA.

The HFA provides low-interest loans for apartment projects to house low- and middle-income people. These projects can produce enormous profits for developers who sell interests in the projects as tax shelters, get federal subsidies to pay back their mortgages and collect fees to build and manage their projects.

New Jersey's independent crime commission, the state attorney general's office and the U.S. attorney's office in Newark began separate investigation of the HFA in February, after a year-long probe of the Agency by The Trenton Times.

Former HFA director William L. Johnston, who had also headed the subsidized housing industry's Washington lobbying group, was forced to resign a month later.

Then, on Thursday, the newspaper reported a private housing consultant's claim that he was forced to hire a close friend of Horgan's as his "co-consultant" on HFA work in 1975, while Horgan was the deputy commissioner of the HFA's parent agency.

The consultant, Joseph Silvestri of Concept Building Industries in Keyport, N.J., told the Times that he made his last $17,500 payment in that deal directly to Horgan in August 1977, a few months before Horgan took over the number two spot in the national party organization.

Horgan does not deny taking the money, but he insisted Thursday that it was part of a legitimate business deal in which he was a partner. Hours after the report was published, White House personnel director Arnie Miller announced that Horgan was resigning his $47,000-a-year consulting contract with Miller's office. Democratic National Committee Chairman John C. White said yesterday that Horgan is not expected to return to his party post.

Silvestri is a major developer of subsidized housing in New Jersey. He claims that Horgan's partner, retired fur salon owner Jack C. Stein of South Brunswick, N.J., warned him in 1975 that he would not get any more HFA loans unless Stein got a share of his fees.

Although Stein had no prior HFA experience, as soon as he was hired the agency suddenly reversed its earlier rejections of three Silvestri projects and moved one project to the top of its priority list for federal funds.

Stein allegedly collected $43,250 in this deal before he died in August 1977. Then, Silvestri says, Horgan told Silvestri he was Stein's partner and he would collect the final payment.

Horgan first surfaced in the HFA affair in May, when The Trenton Times disclosed that he and Arthur Lerner, the son of a powerful Democratic county chairman in New Jersey, had taken control of two Neward housing projects from two nonprofit groups, although the two men lacked the money needed to meet the HFA down payment.

Those projects generated considerable profit for Horgan and Lerner, since they produced tax-shelter investments and were awarded up to $4.8 million a year in federal rent subsidies through the HRA.

Lerner and Horgan had set up a separate company to collect the fees for managing the two projects. Until his death, Stein was a partner with them in these ventures.

In addition to the allegations involving Horgan, State and Federal investigators are probing Trenton Times disclosures that HFA employes took costly gifts from HFA project owners, that federal rent subsidies were borrowed interest-free by developers and that the HFA burned a blind eye toward questionable practices by politically powerful developers. CAPTION: Picture, DANIEL W. HORGAN...says it was a business deal