Housing and Urban Development Secretary Samuel R. Pierce Jr. yesterday defended his performance as the highest-ranking black in the federal government, but he did not try to defend Reagan administration policies before an audience of black government workers.

Pierce's 30-minute speech at the final session of a four-day conference of Blacks in Government (BIG), an association of federal, state and local government employes, surprised some of the 1,000 black federal workers in the audience who had expected him to answer criticism leveled at the administration earlier in the week. BIG officials charged Thursday that the Reagan administration's budget cuts and reductions in force (RIFs) have forced a disproportionate number of blacks out of the government.

Pierce made no mention of the RIFs, but said that the number of blacks in high-level government positions had "dramatically changed" in the last 30 years. Between 1948 and 1961, he said, there were only four blacks in subcabinet positions and one at the presidential staff level. In contrast, he said, blacks got eight of 49 supergrade or Senior Executive Service appointments in HUD in the last 18 months.

"We have come a long way . . . but we still have a long way to go," Pierce said.

"It wasn't much of a speech," Henry (Sherwood) Satterwaith, a government worker from Hyattsville, said afterward. "I wanted him to tell us what the administration is doing for blacks. . . . About the only thing I can see it doing is teaching everyone to look out for themselves. I see a lot more old men selling snow cones on the streets now because they need the work."

Pierce, a former New York corporate lawyer whose low profile in his Cabinet job led senior civil servants in his own department to tag him "Silent Sam," has been criticized for not being a vocal advocate for blacks. But he said yesterday that he endorses job preference for well-qualified minorities and that he exercises his influence to promote civil rights "whenever and wherever I can."

Pierce said the media misunderstood his role last year during a White House debate over extending the 1965 Voting Rights Act. Pierce said he was unfairly criticized at the time because he supported a watered-down version of a tough House bill that Reagan disliked.

He endorsed that compromise bill, Pierce said, because he felt there was an "excellent chance" that Reagan was about to choose an alternative plan backed by Attorney General William French Smith that would have removed the "very heart of the act."

Pierce also said recent charges of racism at HUD were not true. The allegations surfaced in a long, unsigned document this summer that detailed two dozen "case studies" of high-level black officials at HUD who had been removed, reassigned or RIFfed as a result of "both subtle and outright discrimination and racism."

"I will not tolerate racial discrimination in any shape, way or form at HUD," said Pierce.

Pierce said that racist allegations by anonymous sources were "irresponsible." The secretary, however, did not mention that the HUD chapter of BIG did its own investigation of the 22 allegations and determined that they were valid. In response, Pierce formed a special task force in July to investigate the allegations.