AT THE heart of the HUD riddle is the enigma, Samuel Pierce, the former Housing and Urban Development secretary, who after months of silence formally took the Fifth Amendment before a House investigating subcommittee. Pierce is the son of an affluent, achieving black family. His academic record commands respect: honors graduate of Cornell, law degree from Yale, LLD from New York University, where he once taught. He has been a judge; he served as an assistant U.S. attorney in the celebrated Southern district of New York. He was fit for high federal office. Why was a man of such excellent reputation overtaken by a terminal languor in his eight years as secretary of HUD; and why, when revelations about gross mismanagement and corruption in his agency surfaced last May, was he so indifferent about defending his good name? Tales about his ennui abound. He watched a soap opera ("All My Children") every day while underlings tossed millions of dollars to their pals outside and cut deals to provide swimming pools and country clubs for the rich. He made five trips to the Soviet Union, the last place on earth to learn anything about housing. The wanderings seemed yet another declaration that nothing in his immediate surroundings warranted his attention. It is said that he came here and he stayed here because he hoped that Ronald Reagan, in the event that Thurgood Marshall was subpoenaed by Heaven, would name him to the Supreme Court. At the same time, he made so little impression on the president that Reagan once dimly addressed him as "Mr. Mayor." Pierce roused himself to make a congressional appearance on May 25, in which he bragged, in his low-key fashion, that he had had nothing to do with HUD operations. Rep. Christopher Shays (R-Conn.) asked him, unbelievingly, "You never said, I think this is a good program and I think you should fund it? You never said that? . . . In your entire eight years there?" "I don't believe so," Pierce replied. Subsequent witnesses contended that Pierce had indeed intervened every now and then. With Pierce's help, James Watt, who spent so much time raving about government extravagance, milked HUD of millions for his clients. Carla Hills, now the nation's trade representative, successfully got his ear for her client, DRG Funding Corp., a private mortgage lender which pulled such tricks that even the indulgent HUD watchdogs growled. Pierce's lethargy seems to have lifted in the face of personal appeals from the powerful. Did he imagine they would lobby for his Supreme Court appointment? Pierce, a tall, well-tailored figure, kept his head down in the hearing room last week. Much rhetoric was decanted over his head. Chairman Tom Lantos bewailed the absence of television cameras and wanted it understood it was not his idea -- the secretary had invoked an ancient ban rule from the McCarthy days. Other members collaborated in prolonging the affair and robbing it of drama. On-camera silence such as Pierce practices would have brought into focus the sight of a Reagan Cabinet officer becoming the first such official since Teapot Dome to take the Fifth. Members are never content to let events speak for themselves. They all natttered on about how important it was that no one leap to the conclusion that Pierce's conduct was criminal just because he won't talk about it. They also went into great detail about how they felt. Republicans do not expect to pay for the HUD scandal. It is one of those shameful episodes that is completely self-contained. The cavalier conduct of those involved is awkward, because it illuminates a Republican attitude towards public service. Democrats like government for the fun of it. Republicans , however, especially in the Reagan era, made no secret of the fact that public service is for them a form of penal servitude, which has to be endured until the moment when they can dig in at the private sector trough and make a mint "consulting" with the agency they have just left. But it all happened in the Reagan years. Jack Kemp, the new secretary, has acknowledged evil and vowed to "drain the swamp." Rookie Republican Shays is making a name for himself by asking piercing questions. One marginal consequence has been to deprive New Jersey GOP gubernatorial candidate Jim Courter of the consultant services of Gregg Stevens and Roger Stone because their firm had HUD dealings. And the GOP campaign to depict the Democrats as the party of moral squalor was stalled. The latest target, Rep. Barney Frank of Massachusetts, whose dealings with an odious male prostitute have brought cries for his resignation, sits on the committee. His ability to make those slashing summary ethical-political judgments has been curbed. But he is no pariah. The ranking Republican, Buz Lukens, was convicted of seducing a minor. They are both under investigation by the House ethics committee. Mary McGrory is a Washington Post columnist.