Senior career civil servants in his own department hae tagged him "Silent Sam." U.S. News & World Report called him the Cabinet's invisible man. And in a meeting at the White House a few days ago, President Reagan momentarily mistook him for a mayor.

Samuel R. Pierce Jr., 58, the black man from the Park Avenue law firm and various Fortune 500 corporate boards, has all but blinked off the screen since becoming Reagan's surprise choice as secretary of Housing and Urban Development.

Stilted and tightly wrapped in his few public appearances and a mute voice in the big Cabinet debates over national policy, this tall and broad-shouldered man is, nonetheless, being given credit for ability to hold his own in the gritty backroom bargaining of Reagan Washington. His department has taken deep cuts but he has gone toe to toe with David Stockman and won a few rounds, preserving the low down payment FHA home loan program and funds for renovating public housing and revitalizing downtowns.

Throughout his life, Pierce has operated differently in the white world from the more familiar charismatic and media-conscious civil rights leaders. He seems to prefer bargaining behind the scenes to the glare of the spotlight.

But there is a question of how well this style will serve him in this town, where visibility and public image often determine clout.

Mayors and career civil servants have found Pierce to be cold and short but the diverse collection of Washington lobbyists with interests in his department are grateful for the battles he has fought for them.

Pierce himself, in a rare moment of ease during a recent interview, recalled with relish the battle he and Stockman had over FHA. The budget chief wanted to begin passing down FHA and eventually place a ceiling or "cap" of about $10 billion on the loans it could insure.

"I said no. That one we would have to go to the president on because FHA, in my opinion, had been one of the great programs of this department," Pierce recalled.

He and Stockman went to the mat on the issue. "We talked and we walked together and saw the light together and we left that one at the $35 billion cap." Pierce said chuckling.

He leaned back in his chair and burst into the laughter of a memory so satisfying that he told the story again.

"Anyway, Stockman wanted it to go down to about 10 [billion dollars]. He wanted to cut it down over a couple of years. So I said, 'No. If you want to go that way, I can tell you right now, we're going to have to go see The Man.'"

Pierce is no stranger to Washington or the boardrooms of corporate power. Before Reagan appointed him housing secretary, he was a $280,000-a-year partner in a powerful New York law firm, a director of General Electric, Prudential and U.S. Industries, among others, and a governor of the American Stock Exchange. At various times, he has been a judge, labor mediator, prosecutor and high-ranking civil servant.

During the Nixon administration he was general counsel at Treasury and then Secretary John Connally's action man in negotiations over the federal bailout of Lockheed Aircraft. In the early 1960s, he represented the late Rev. Martin Luther King Jr. in a case before the Supreme Court that resulted in a landmark decision.

It was Alfred Bloomingdale, Reagan intimate and Diner's Club founder, who called him about serving in the administration. Pierce had been a lawyer for the Diner's Club, and Bloomingdale is among the diverse collection of his friends and admirers. They range from former Treasury under-secretary Charls Walker to National Urban League president Vernon Jordan and Watergate burglar G. Gordon Liddy and they are effusive in their praise.

On at least on occasion, Pierce has found the admiration embarrassing. In 1964, when the FBI was wiretapping Rev. King and attempting to discredit and topple him, assistant FBI director William Sullivan began to worry about what would happen to the "Negro people" once that mission was accomplished. In a memo written at the time, Sullivan talked enthusiastically of Pierce after seeing a list of his achievements, saying he had "all the qualifications of the kind of Negro I have in mind to advance to positions of national leadership."

Pierce said he only learned of these intentions years later when the FBI's campaign against King was investigated by a Senate committee. He said he found the effort and its disclosure embarrassing, not least because he had considered King a personal friend.

The FBI also apparently did not know that just two days before Sullivan's memo, Pierce had argued, at King's request and for no fee, the appeal of a libel judgement won by the police commissioner of Montgomery, Ala., against The New York Times and four clergymen for an allegedly defamatory advertisement seeking contributions for King and the civil rights movement in the South. The court ruled in favor of the Times and King's allies in a landmark libel decision.

Pierce has frequently been on the edges of the civil rights movement, raising money for both the Urban League and NAACP. He and baseball great Jackie Robinson were among the founders of the Freedom National Bank in Harlem, where he and his wife Barbara, a physician and the daughter of a widely respected surgeon at Harlem Hospital, lived for a number of years before moving to the Upper West Side off Central Park. They have a daughter.

Throughout his career, as he has ascended the ladder to the upper reaches of the white establishment, Pierce has always kept a foot in the world of blacks.

After graduating from Cornell law school, where as an undergraduate he had been a star halfback and Phi Beta Kappa, he joined the New York Young Republicans, whose membership included well born and ambitious young men like John Lindsay. At the same time, his was a familar face around the Harlem clubhouse of the Square Deal Republican Club, where there are always a pot of pinto beans and pig tails simmering on the stove.

Nevertheless, friends of Pierce found the FBI's scheme for enthroning him as the national black leader succeeding King preposterous. For all his achievements, Pierce could never get the voters of Harlem to elect him judge.

Gov. Nelson Rockfeller appointed Pierce in 1959 to a seat on New York's old Court of General Sessions and he served two years before having to run for election. He tried twice and lost both times. His overwhelming disadvantage was that he was a Republican running in solidly Democratic Harlem. But friends recall with fond amusement that there were other problems.

Chuck Stone, now senior editor of the Philadelphia Daily News but in the late '50s editor of the New York Age, a Republican-leaning black newspaper, remembers going out on the campaign trail with Pierce. They met at Pierce's apartment in the then-fashionable Riverton in Harlem. When the elevator reached the lobby and they and four others got out, Stone told Pierce, "You just made your first mistake. You didn't speak to anybody."

"They didn't speak to me," Stone remembers Pierce responding.

"Yeah, but they aren't running for judge," Stone said.

When they got around the corner, Stone recalls, Pierce marched over to a small group of winos and junkies, shook their hands and talked about his candidacy. Stone afterward told Pierce that this was not a group likely to troop to the polls on Election Day. Beter to speak to the folks in the elevator.

Stories like that are being told around town with titters to explain the extraordinary sobriety of the HUD secretary.

Another of Pierce's old friends, lawyer William T. Coleman Jr., retorts, "he has as much charisma as the Treasury secretary." Coleman also argued against making too much of the fact that Pierce is black, remembering just how much it rankled when newspapers referred to him as the black transportation secretary when he served in Gerald Ford's Cabinet. Credit Pierce with special sensitivity because he is black, but give him credit for his illustrious background and legal career, Coleman admonished.

But the fact is that Pierce -- as overseer of 15,500 federal employes and a $13 billion annual budget -- is the most important black man in a government deeply suspected by other blacks and when the influence of civil rights leaders on national policy is at its lowest in two decades.

Pierce can talk about thrashing out a $300 million dispute with Stockman over public housing funds at a Cabinet meeting. By contrast, civil rights spokesman Jesse Jackson has been fighting for months with the Labor Department to hang on to a $2 million Carter administration grant to establish a Washington research office. The funds represent a third or more of the annual operating budget of Jackson's Operation PUSH. Complaining that the Reagan administration is retaliating for his support of Carter's reelection, he has sued to keep the money. Pierce has been negotiating a settlement.

Many blacks seek him out for help, he said, on matters that range far beyond the jurisdiction of his department. He indicated in the interview that his first obligation is to be housing secretary but, yes, he does also feel some responsibility toward fellow blacks:

"Let me put it this way. Being black, you want to try to help blacks whenever you can. That's a natural thing, because I think there is a sort of brotherhood with black people that's a little deeper, I think, than with most whites."

Just what that feeling of responsibility means he did not clearly define. One thing he is clear on -- he alone cannot be the leader bringing blacks into the national consensus. "That's too big a role for one man to play." That blacks are outside of the consensus is the result of "the years in the United States. That's everything all put together. And it's hard to change a situation like that."

Change will come about with integration, he said. Other blacks may have grown skeptical about the possibility and ultimate desirability of integration, but on this subject Pierce is fervent. He grew up in integrated Glen Cove, Long Island, a community of blacks, white Portestants, Italians, Irish who all got along, he said.

"I think integration is so important because there you get to know your fellow man. You get to be able to feel him, to understand him, to get to know that he's just a person just like you are."

He said he has known many blacks who did not grow up among whites and confessed as adults to begin uncomfortable in integrated settings.

"I feel at ease. And I think you have to have that feeling that you're ready to move in and go and participate and then it doesn't bother you. And I think that kind of feeling has to come over the masses of people and that's why I think integration is so good. Because I don't think there's anything that gives it to you in a better kind of way than just being integrated."

That blacks would emerge in public opinion polls as the single identifiable group fiercely opposed to Reagan and his economic policies does not surprise him, nor is it a matter for which he has any ready answers.

He uses the Congressional Black Caucus as the example for the point he wants to make. "They believe in government helping and being the guiding sponsor to many things. They believe that the government has a responsibility to the people, much less than the people have a responsibility to the government. So, when you believe in [Black Caucus] philosophy, it's hard to believe in Reagan."

He feels differently?

"I feel differently, yes," he says softly.

He then bursts into a speech he makes often in public.

"I feel that this country is in very serious economic trouble. I more than feel it. I know it. Our country is going to go bust similar to what happened to Germany after World War I, Very similar."

Pierce's personal portfolio of investments indicates that he sincerely believes this. His holdings were in numerous small "odd lots" of Triple A stock, assets assembled seemingly to survive an "economic Dunkirk." Leslie J. Silverstone, a stockbroker and regional vice president for Dean Witter Reynolds here, described Pierce's portfolio as superconservative.

During the Depression, blacks in this country turned in droves from the party of Abraham Lincoln to the party of Franklin Roosevelt. But one man who did not desert the GOP was Samuel R. Pierce Sr. of Glen Cove, Long Island. Friends of the housing secretary say that to understand him, you must know about his father.

The elder Pierce moved to New York from Portsmouth, Va., around the turn of the century. He worked as a handyman for a family in Brooklyn, then as a locker room attendant at the Nassau Country Club on Long Island. After a year or two, he persuaded the rich men in the club that they needed a laundry and drycleaner and haberdashery shop on the premises, which he established, owned and operated. He expanded the business into the town nearby and over the years amassed holdings in real estate and the stock market.

He and his wife had three sons, Samuel Jr. the oldest. Bert, the youngest, remembers how the elder Pierce would come home in the evenings, bathe, put on his smoking jacket and talk to his three sons in what they called "skull sessions."

"I want you to be somebody," son Sam recalls him saying. "If I wanted you just to make money I would bring you into my business." Instead, he counseled them to get a good education and work hard.

"He just imbued in all of us: reach for the stars but keep your feet on the ground and don't forget from whence you came," Bert Pierce remembered with emotion. He is not the administrator for the Battery Park City Authority in New York. The middle son, Chester, is a professor at Harvard.

Throughout the years, Bert said, the three brothers have frequently repeated to each other another of their late father's maxims: "If you can't make 100, make 99."

Samuel Pierce Jr. now finds himself arguably with his toughest assignment. A private man, he faces many public demands. An ardent believer in integration, he finds himself in an administration moving to end many of the enforcement procedures employed to achieve that. He says he wants to manage the Department of Housing and Urban Development efficiently, from field office to headquarters -- a goal that had eluded all his predecessors. His budget is shrinking as the demand for federal resources for housing increases.

In the interview, he spoke only in general terms about what initiatives he intends to take.He said the rent supplement program for the poor and working class has proved too costly and must be altered or replaced. He wants to know whether housing vouchers might work better and is interested in finding a way for the poor to own the buildings they live in. But these are ideas to be explored by a presidential commission on housing which was named this week.

He acknowledges that there are serious problems with public housing projects in many big cities. Even though he was able to appeal successfully to Reagan to restore $300 million for renovation that Stockman had cut, he said there will not be enough money to do all that needs to be done.

"It's going to be a tough situation, no doubt about that," he said.

The more he learns about his department and its problems, he said, the more he has the feeling of running around with a net trying to catch bombs as they fall from the sky.