As controversy engulfed him last week, the chairman of the U.S. Commission on Civil Rights recalled the days of lunch counter sit-ins in an effort to explain why President Reagan had just fired half the commission's members.

"Remember when some restaurants wouldn't serve you if you were black?" asked Clarence M. Pendleton Jr. "That was discrimination. That's what the commission is supposed to fight.

"But civil rights stops at getting you in the door," said Pendleton, who is a Reagan appointee and will be the only black on the commission if Reagan's new nominees are confirmed. "It isn't getting a free meal when you go in the restaurant. Food stamps is social policy. They the commissioners Reagan seeks to replace are constantly arguing that any cut in special programs like food stamps is evidence of our, the administration's, opposition to civil rights.

"That's a false premise. Civil rights says if the government passes a social program everyone will have equal access to it. But nothing says we must have social programs to prove we believe in civil rights."

This is where the Reagan administration finally comes down on civil rights. It is the rationale for the administration's positions on issues ranging from college admissions and steel mill hiring quotas to busing. It is also the reason the three commissioners were fired last week: They disagree with the president.

The Reagan administration, including Pendleton, essentially looks on civil rights as a problem solved by laws passed mainly in the 1960s making it illegal to deny blacks and other minorities equal access to public accommodations, schools, jobs and housing.

There is little or no governmental obligation to go beyond these laws to help assure equal outcomes in the society, the administraiton believes. On the basis of that conviction, the administration has argued against school busing to end segregation and more vigorously against affirmative action programs, the use of racial quotas in higher education, hiring and government contracting.

Mainly over affirmative action the White House pressured the commission's former chairman to resign last year and twice tried to name new commissioners more to Reagan's liking. It also demanded unsuccessfully that the commission clear all statements and studies with the Office of Management and Budget, and it has tried to cut the commission's budget.

When Reagan fired the three commission members last week the civil rights community was outraged.

"When that commission was created in 1957 it was meant to be a burr under the saddle of the president and the Congress on racial problems we've often preferred to sweep under the rug in this country," said Joseph L. Rauh Jr., general counsel of the Leadership Conference on Civil Rights.

"Now, because Reagan does not want the goad, he is attempting to destroy the very purpose of the commission," Rauh said. "Don't tell me that cuts in food stamps aren't a civil rights issue. Food stamps are a form of recompense for past discrimination against minorities, blacks, Hispanics, whose parents didn't get enough to eat, whose parents did not have the chance to grow and learn like white children. That is a racial issue."

In a similar vein, Benjamin Hooks, executive director of the NAACP, said that the administration's professedly colorblind policies are undermining civil rights.

"You cannot have one man who has been fed good milk and cream and . . . take his brother who has been denied all those things . . . say 'You are now equal to run the race of life. Don't look to us to compensate for past inequities.' "

The White House, which has been seeking to soften its sharp image on civil rights, was taken aback by the furor over the firings at the Civil Rights Commission. Among other things, the outcry came just as the Supreme Court in effect rebuked the administration on another celebrated civil rights issue by ruling against it on tax-exempt status for private schools that discriminate.

"We fumbled the ball," said one senior administration official. "We didn't expect the Supreme Court decision, and we have a new personnel director John Herrington who pushed the appointments through before we had thought out the political impact."

The administration, and conservatives generally, think that the Civil Rights Commission in recent years has been almost meddlesome and disorganized.

It has investigated such subjects as the feminization of poverty (even as the administration has been fighting to overcome the so-called gender gap whereby women have tended in recent years to vote Democratic). It has gone into the silicon valley in northern California to see how minorities and women are faring at high-technology companies. It once did a study of discrimination in television shows, White House officials like to recall.

Yet, while it was busy with these issues, which its conservative critics regard as peripheral, its report on the value of extending the Voting Rights Act was not prepared in time for the start of hearings on the subject in the last Congress.

"The fact that the commission seeks other activities to justify its existence doesn't put it in a class by itself," black conservative Walter Williams wrote recently. "After all, the March of Dimes mandate was to eliminate polio. Polio is gone, but the March of Dimes lives on with a bigger budget."

Other conservatives have equated the commission's advocacy of domestic spending programs with the activities of interest groups.

Chester E. Finn, professor of education and public policy at Vanderbilt University, wrote in The Wall Street Journal in March that the commission was "clothing even its most hyperphagic forays to the public treasury in the raiments of social justice and cultural legitimacy not easily donned by the dairy industry, the ship builders or the investment bankers."

"The commission's appetite for generously funded aid and service programs appears insatiable," Finn wrote. "But by endeavoring to throw the apolitical mantle of civil rights over such interest group advocacy by impuging the motives of those who may place greater weight on other interests . . . , by smiling at discrimination and favoritism when those same clients stand to profit and by hinting that unpleasantness awaits those who don't share its priorites, the commission has gone a long way toward transforming democratic principles of universal applicability and great moral power into cynical gamesmanship."

"The commission is an octopus; it has its hands in so many things," said an administration official. "It's got its hand in everything looking for problems. That's not its job. We don't need that."

The best recent example of this interventionist instinct, and why the administration dislikes it, may have come last month when the commission made public a letter to Reagan complaining that the administration was keeping the Equal Employment Opportunity Commission from taking a position in favor of racial quotas in an affirmative action court case involving the New Orleans Police Department.

Five commissioners, including Vice Chairman Mary Louise Smith, who is a former Republican Party chairman and a Reagan appointee, signed the letter. Only Pendleton did not. He later wrote a separate letter to Reagan calling his fellow commissioners "presumptuous." That was five weeks before Reagan fired three of them.

The White House had tried to replace the three commissioners, Mary Frances Berry, Rabbi Murray Saltzman and Blandina Cardenas Ramirez, last summer, but the Senate never voted on their replacements.

Earlier, Reagan also had nominated a little-known Philadelphian, the Rev. B. Sam Hart, to the commission, but withdrew his name after Hart made controversial statements, including one suggesting that he did not think homosexuals deserve special civil rights protection.

This year the administration proposed extending the commission's charter, which is scheduled to expire this year, for 20 years, but only if the commissioners' terms become fixed. If that proposal were law now, the Reagan White House would be able to replace all the commissioners.

A House subcommittee said that it would agree to the plan if some current commissioners were guaranteed appointments. The White House refused.

"We were outraged then that he Reagan wanted to replace the commissioners with people who had the same views he holds. And we're outraged now that he is trying to do it by firing them," said Rep. Don Edwards (D-Calif.), chairman of the House Judiciary subcommittee on civil and constitutional rights.

"The independence of the commission is its reason for being. It has irritated Lyndon Johnson and Jimmy Carter as much as Ronald Reagan. They need to be independent," Edwards said.

Jerris Leonard, who was assistant attorney general for civil rights during the Nixon administration, said, "We had as much trouble with the commission as William Bradford Reynolds the current civil rights assistant and this administration are having.

"The commission always wants to be on the cutting edge. And that's all right as long as they don't get demagogic about it. They say, 'You guys are doing nothing' instead of saying, 'There is some good, but you could do more.'

"The budget issue, if you look at it, is always going have a greater impact on minorities. But the other question they don't talk about is whether the programs work. We have lots of evidence that not all programs work."

But quotas are the main policy point on which the commission and administration disagree. And defenders of the commission point out that it is the Supreme Court, not the commission, that has ruled that quotas may be used to remedy past discrimination.

"In the 1970s, the Supreme court has let quotas stand as a tool for use in redressing discrimination," said Eleanor Holmes Norton, a law professor at Georgetown University who headed the Equal Employment Opportunity Commission under President Carter.

"But," she added, "in Reagan's 1950s view of America, discrimination is only when a black is denied entrance to a school, denied a job. He doesn't look at the effects of historic discrimination on that person when he finally gets in the school. The commission he is complaining about wouldn't be worth much if it only looked at immediate causes. It is supposed to look deeper and detect the roots of things the nation may not detect."

Reagan's three new nominees--Morris Abram, former president of Brandeis University and long active in the civil rights field; John H. Bunzel, a senior research fellow at the Hoover Institution at Stanford University, and Robert A. Destro, a law professor at Catholic University--all share his view of affirmative action. But all three have excellent credentials; none is assailable in the way Hart was.

Reagan is accused of seeking to stack the commission, but White House officials note that Carter considered appointing Abram to the commission in 1979 and decided against it when he found out that Abram opposed quotas.

"There is a great big argument in this country over whether the Constitution is colorblind," said Walter Berns of the American Enterprise Institute.

"And I think as a political scientist that the president of the United States is entitled to have a civil rights commission that agrees with him on the shady areas of that big argument. I think the Constitution is intended to be colorblind, and it does not take into account benign or malevolent quotas . . . . A malevolent quota kept Jews out of Harvard. A benign quota puts unqualified people in Harvard."