The civil rights case that bedeviled Congress and the Reagan administration last year -- the controversy over tax benefits for discriminatory religious schools -- came before the Supreme Court yesterday and several justices suggested they too will be walking a fine line in resolving it.

The largest crowds since the Watergate tapes and reverse discrimination cases of the 1970s turned out for what some regard as the civil rights movement's most celebrated confrontation with the Reagan administration. Lawyers from all over the country, religious officials, fundamentalist children in their green and white Christian school uniforms and out-of-town journalists competed for admission.

In the most unusual moment of the day, those who got in saw a lawyer without a client, William T. Coleman Jr., rise to act as a special advocate in defense of the Internal Revenue Service's authority to deny the lucrative tax exemptions to institutions that practice racial discrimination.

Coleman, a former Cabinet secretary and one of the country's most prominent black lawyers, was appointed by the court last term to argue the position abandoned by the Reagan administration when it switched gears last year in the IRS case.

They heard William Bradford Reynolds, the government's top civil rights official, open his argument with an unusual declaration, clearly addressed to a broader audience, of the government's overall "commitment to the elimination of discrimination."

And they heard lawyers for Bob Jones University and Goldsboro Christian Schools, which originated the cases heard yesterday when they were denied tax exemptions for practicing discrimination, defend their right to do what is "dictated by the Scriptures."

At issue was whether the IRS, without explicit authorization from Congress, could decide that discriminatory schools were unworthy of tax-exempt status. The IRS has been doing this since 1970, at a cost of millions of dollars in taxes and contributions to the schools.

Several of the justices made their concerns known. John Paul Stevens suggested that if the government had too little authority to deny tax exemptions even "Fagin's school for pickpockets" (in Charles Dickens' classic "Oliver Twist") would be entitled to tax-exempt status.

"But a school for pickpockets is not organized for educational purposes," responded William G. McNairy, the lawyer for Goldsboro.

"Why not?" said Stevens.

"It's organized for criminal purposes," McNairy said.

"It's still teaching something, isn't it," said the justice, drawing laughter from the audience.

Too much authority, said Justice Lewis F. Powell Jr., could be used to penalize an otherwise tax-exempt pacifist or socialist organization the government disliked. Where does the authority end, he said. "What limits it?"

"Could the IRS make the same determination and deny tax exemptions because of sex discrimination?"

"That would be a different question," Coleman responded.

"Why?" asked Powell.

"We didn't fight a civil war over sex discrimination and pass an amendment to the Constitution against it. No one can say it's as fundamental in our history . . . . You just can't compare this with any other activity," Coleman said. "This [the public policy against race discrimination] is the one policy that is crystal clear."

Bob Jones University, of Greenville, S.C., and Goldsboro, of the town of the same name in North Carolina, have maintained racially restrictive admissions policies since their founding. In 1970, in an attempt to discourage the growth of the "segregation academies" that multiplied in the wake of court-ordered desegregation, the IRS instituted its policy denying or revoking exemptions otherwise available to all nonprofit charitable and educational organizations.

The policy was based not on any act of Congress but on a traditional legal concept of a charitable enterprise: it must be consistent with law or fundamental public policy, in this situation, the law and policy against racial discrimination.

Bob Jones and Goldsboro were ruled ineligible for the exemptions. They thus became obligated to pay thousands of dollars in federal taxes and lost the right to receive tax-deductible contributions. They have been fighting the case in court, unsuccessfully, ever since.

Their argument is two-pronged. Congress never authorized the broad power being exercised by the IRS, they say. And even if it did, a discrimination based on religious belief is tamper-proof under the First Amendment's guarantee of religious freedom.

William B. Ball, one of the country's most prominent lawyers in cases involving religious liberty, began the arguments yesterday representing Bob Jones. Ball has also been involved in equal rights litigation and has expressed discomfort at being identified in this case with what many perceive as an anti-civil-rights cause. Yesterday, he quickly placed distance between himself and his client.

"Bob Jones' theology may not be yours," he told the justices, "and it certainly isn't mine. But it is dictated by Scripture." And the revocation of its tax exemption because of its religious belief constitutes "a threat to the religious liberty of everyone."

The justices, in an unusual display of restraint, allowed Ball to talk without interruption for nearly 10 minutes. McNairy followed, making the more technical argument that the definition of "charitable" used in denying the exemptions was too broad. "The Internal Revenue Service granted exemptions to schools for 50 years without regard" to their racial policies, he said. "Then, on June 10, 1970, it announced in a press release that it would no longer grant them."

" . . . These are political questions that have been allocated to Congress," McNairy said, "not to the commissioner of the IRS."

Reynolds, assistant attorney general for civil rights, then began his statement. A target of criticism by civil rights groups since the day he was appointed with experience largely in business law, Reynolds' role in the Reagan administration's reversal of the tax-exemption policy fueled a near revolt among some lawyers in his division. Since then, in speeches, lunches at newspapers and letters to the editor, he has been defending himself and the administration.

"The U.S. government has no tolerance for racial discrimination in schools, public or private," he told the court. "We are committed to the elimination of all vestiges of racial discrimination.

"These cases do not call into question that commitment," he said. "Bob Jones and Goldsboro unquestionably run afoul of U.S. civil rights policy."

It is true, he said, that the administration "changed its position" in the cases. But it did so for a "plain and simple reason." After studying the history of the IRS decisions, the administration concluded that there was "no evidence" of a grant of such broad authority to the IRS.

Coleman then rose and took on the other three lawyers. Reynolds, he said, "is wrong. He's just wrong." Congress has known about the IRS policy for a decade, he said. It has been upheld by lower courts, debated on Capitol Hill and aired widely for 10 years. Yet Congress "made no change" in it, he said.

That constitutes a form of ratification of the IRS policy, he said.

Coleman's exchanges with the justices were unusually informal. Interrupted on one occasion by Stevens, he told the justice to "hear me out. Hear me out."