Housing and Urban Development Secretary Samuel R. Pierce Jr. presented the administration's fiscal 1984 program for his department to an openly skeptical House subcommittee yesterday, and he came out swinging, telling one member to "stop playing like a little boy" and snapping at another, "You just don't know what you're talking about."

Subcommittee Chairman Henry B. Gonzalez (D-Tex.), who had opened the hearing by wondering out loud if Pierce would sabotage a mortgage foreclosure relief program if Congress passes one, chided him at one point for an "unacceptable performance by a witness."

Pierce had gone before the housing subcommittee of the House Banking Committee to outline HUD's legislative agenda for the coming year. In a lengthy prepared statement, he cited programs and initiatives that he said represented "important progress" toward improved housing for all Americans.

He admitted that "unmet problems and needs remain." But he argued that the administration's proposals--which feature a housing voucher plan to replace most rent subsidies and low-income housing production programs--will "help poor people afford good housing in the broader community."

In addition to the vouchers--or housing payment certificates, as HUD prefers to call them--Pierce said the administration also is seeking:

* $150 million to rehabilitate about 30,000 rental units.

* A three-year, $3.5 billion reauthorization of the Community Development Block Grant program, and changes in the program to allow communities with specific need to use the money to build new housing.

* $196 million in fiscal 1984 and $440 million in each of the following two years for the Urban Development Action Grant program, and changes in the program to facilitate the participation of small cities.

* $12 million to expand the Urban Homesteading program.

* $76 million for improved housing programs for Indians.

The secretary also described proposals to alter the way public housing is funded--designed to modernize some of the stock and improve the way it is managed--and listed changes in FHA programs.

But as he ticked off the numbers, it became clear that many members of the panel, particularly Democrats, regarded them as inadequate.

Rep. Charles E. Schumer (D-N.Y.) said that "the budget that you've submitted reminds me of the opening scene of 'Annie Hall,' where two people are seated at a diner and the first one says, 'The food is terrible here.' And the second responds, 'Yeah, but the portions are small.' "

Rep. Mike Lowry (D-Wash.) attacked Pierce for resisting proposals for a foreclosure relief bill on grounds that the expense was unnecessary and would send the wrong signal to financial markets, thus possibly driving up interest rates.

He said the three-year cost of the military budget is "more than a thousand times" that of the foreclosure bill. "Are the financial markets so out to lunch that they think a three-year $760 million authorization is causing the deficit . . . ?" he asked.

Pierce replied that it was not the one bill but the pattern of "cut the budget but don't do it to me" bills that would alarm the markets.

"But the financial market is going to be affected," Lowry persisted, "by a $700--"

"I didn't say that, Mr.--," Pierce interrupted.

"Yes, you did."

"No. I didn't. I said--"

"You said--"

"Look, look, stop playing like a little boy. Listen--"

"Hey, I'm not playing like a little boy, Mr. Secretary," Lowry shot back. "I wasn't playing like a little boy. The defense budget isn't a little thing."

At that point, Gonzalez gaveled the men to silence to "insist that we have decorum" and to chide Pierce for a remark attacking Lowry rather than debating the issue.

But the secretary was clearly annoyed and not in a conciliatory mood. When Rep. Bruce A. Morrison (D-Conn.) accused him of "wearing rose-colored glasses about the reality out there" and suggested that the voucher program would end up subsidizing substandard housing, Pierce interrupted, "I totally disagree with that, and I think you just don't know what you're talking about."