When a black man with $675,000 to buy a house has to sue to get people to sell it to him, then you know that government affirmative-action programs are still desperately needed, said outgoing Health and Human Services Secretary Patricia Roberts Harris in a passionate defense of special programs for minorities.

Harris, who usually is cool and collected during interviews, was anything but that when asked whether, with the Reagan administration, the time has come to rethink affirmative action.

"White people are still maintaining the position of the inferiority of black people . . . that black people are so inferior that they may not associate in the school system and "housing with whites, Harris said.

"Don't play games with me by gerrymandering the [school] districts. . . . Do not use bombs [or] gentlemen's agreements to keep me out of neighborhoods."

Harris said that when she got out of law school 20 years ago, "I was never interviewed by a law firm, not one," although she was first in her class. Being a black and a woman was simply too much for them. And even 10 years ago, when she became a senior partner in a prominent law firm here, "I was the first black and first woman" to have such a position.

And even today, "I do not believe that if I wanted to buy a house in some sections of this country, the real estate agent would tell me what's really available."

In an interview as she prepared to surrender the reins of the $200-billion-a-year agency to Sen. Richard S. Schweiker (R-Pa.), who President-elect Ronald Reagan will nominate as her successor, Harris also said:

A report released two years ago by the department, asserting that there might be as much as $7.4 billion a year waste, fraud and abuse in its programs, did "terrible damage" to poor people and was grossly misleading because it led everyone to think that there was actually $7.4 billion in welfare and other fraud; whereas, the report alleged only $1 billion in fraud; the remaining $6 billion was simply estimates of how much might be saved if policies and laws were changed. Harris, who wasn't HHS secretary then (Joseph A. Califano was), said, "I sat in front of the TV set and watched Joe and said 'Oh, my God, we are never going to recover from this.'" She said as far as she knows, nobody intended the report to sound the way it did, including Califano, but since then, "it has been misused by everyone who cites it."

Of all HHS programs, she fears that the one most in danger from the budget-cutting proclivities of the Reagan team may be Aid to Families with Dependent Children, the welfare program most people think of. Some of the Reagan people seem to favor very wide discretion for the states to revise program parameters or dollar caps on the program regardess of need. People must not forget, she said, that two-thirds of the beneficaries of this program are needy children.

Although Caspar Weinberger, Reagan's choice as defense secretary, has been labeled "Cap the Knife" for his alleged budget-cutting during the last GOP administration when he headed Harris' department, it actually grew more, " by 28,000 employes," during his tenure there than at any other time.

One of the reasons for President Carter's defeat was his failure to communicate adequately to people -- "he just didn't talk about it" enough -- how much human services programs in housing, welfare and health were directly helping them. For example, Harris said, special vaccination and anti-measles campaigns during the Carter administration would have saved thousands of children from illness and crippling effects, but that fact hadn't adequately been driven home during the election. "I don't think he made a health speech during the election." Another factor, Harris conceded, was the economy. No matter who, if anyone, is responsible for economic problems, she said, the incumbent is blamed and falls victim to an attitude of "throw the rascals out."

Schweiker, her successor, is an exceptionally knowledgeable man on HHS programs because of his Appropriations subcommittee experience in the Senate. So "he will not make mistakes out of ignorance." Her experience has been that the people most likely to attack and slash the department's programs are those who don't really know them well.

Her biggest defeat in head-knocking intra-Cabinet and White House policy conflicts was her inability to reorganize the Department of Housing and Urban Development, which she headed before going to HHS, into a general agency to assist in general economic and urban development. Her proposal intended to incorporate into HUD the Commerce Department's economic development program and certain rural programs now in the Agriculture Department. She was blocked, she said, when "the White House staff took a position consistent with the Commerce Department [then headed by Juanita Kreps] which saw this as elimination of their identity."

She is proud of her success in getting the Carter administration to drop proposals, developed while Califano was secretary, to slash certain Social Security benefits in order to save money. She said worked to get the proposals dropped "with a little help from Wilbur Cohen," a former secretary who organized a massive coalition of organizations of the aged and disabled to oppose the cuts.

One of the things "that has shocked me is the fact that the only way government officials can communicate with people . . . is through the media and I find the media idiosyncratic. If your message is a message of failure, a message of corruption, that's the only thing gets out there." The media reinforce the message of failure, she said. Invariably, when they refer to school busing, they use the phrase "forced busing," she said, adding the media rarely report good news like the fact that food stamps have virtually eliminated malnutrition in the United States or that communicable diseases are being eliminated.

Although she strongly favors affirmative action, she's not talking about giving jobs to incompetents and "turkeys" merely because they are black, Hispanic or female. "People who are employed must be of the caliber to perform the job," like Herb Doggette, a black who would never have risen high in the Social Security Administration, although he is "an outstanding administrator," were it not for affirmative action. Once, she recalled, people brought her a black male for a job and "they said he was wonderful." But actually, "he was a real turkey." So she warned her aides, "Don't you ever send me somebody you know to be a turkey," and drove home her point that that's the way it should be.