Despite a drastic decline last month in housing starts and gloomy predictions for the rest of the year, federal regulators argued yesterday against singling out the housing industry for a quick-fix stimulus.

Federal Reserve Governor Charles Partee warned the Joint Economic Committee that subsidy programs such as those used in the 1974-75 recession would provide some support for housing activity in the short run, but "federal borrwoing to finance these programs would tend to put further upward pressure on market interest rates and could thereby intensify the problems being experienced by thrift institutions, he warned.

He continued, "Short-run solutions designed to aid the mortgage and housing markets will not go to the core of the problem facing these and other sectors of the economy. In order to obtain lasting improvement the inflationary process must be halted."

The administration is expected to announce support today for a revision of Section 235 of the home ownership assistance program that would allow buyers with incomes of between 120 and 150 percent of the median in their area to purchase homes costing about $60,000 at 11 percent interst. The $135 million program, which would subsidize up to 100,000 unites, already has been appropriated and therefore doesn't represent a new expenditure.

Partee stressed that the Fed already had favored housing by exempting mortgages from its credit restraints. He also emphasized that the Fed will allow banks to exceed their 9 percent credit growth, limit if the loans are made to small business, builders and agricultural businesses. A clairifying letter soon will be sent to banks, he testified.

Yesterday's hearing was the first of what are to be quarterly sessions on housing and the economy before the JEC. Chairman Lloyd Bentsen (D-Tex.) noted wryly that it got off to a "rather inauspicious beginning." He was referring to the 20 percent prime rate, the 18 percent inflation rate and the just-announced plunge in housing starts.

The number of units begun in March fell to a seasonally adjusted annual rate on one million, 22 percent below the 1.3 annual rate in February 1980 and 42 percent below the rate of 1.8 million in March 1979. Moreover, housing permits -- the indication of future starts -- dipped to a seasonally adjusted annual rate of 941,000. That rate is 18 percent below February 1980 and 42 percent below a year ago.

Sen. Bentsen termed the decline disastrous. Rep Clarence J. Brown (R-Ohio), ranking minority member, said "the present situation is one in which mortgage lenders can't lend, home-builders can't build and homebuyers can't buy."

Jack Carlson, chief economist of the National Association of Realtors, observed that the housing industry has an unemployment rate of 33 percent and a one-third loss of income.