A company that defeats in court a complaint of racial discrimination in employment can collect legal fees only in highly restricted circumstances, the Supreme Court ruled yesterday, 8 to 0.

The complaint must be shown to have been "frivolous, unreasonable or groundless," Justice Potter Stewart wrote for the court in a case involving an employer who successfully defended against a complaint brought under the civil Rights Act of 1964.

The employer was the Christiansburg Garment Co., of Christianburg, Va., southwest of Roanoke.

The company laid off Rosa Helm, a black, for a month in 1968, and she complained to the Equal Employment Opportunity Commission that the firm had engaged in racial discrimination in violation of the 1964 law.

Two years later, the EEOC notified her that it had tried but failed to conciliate the case and that she now was entitled to sue. She didn't.

Two years after that, in 1972, Congress amended the law to authorize the EEOC to prosecute in its own name that were "pending with the commission" on the effective date of the amendments.

The commission then sued the firm. But a federal judge ruled for Christiansburg Garment on the ground that the Rosa Helm charge had not been "pending" when the amendments took effect.

The company sued for attorney's fees, saying that the rights law allows them for successful plaintiffs and therefore should allow them for successful defendants, except where special circumstances would make this unjust.

In the opinion for the Supreme Court, Justice Potter Stewart termed the company's position "untenable," partly because "the plaintiff is the chosen instrument of Congress to vindicate" an equal-rights policy to which it assigned highest priority. And when a federal court awards fees to a prevailing plaintiff, it is awarding them against "a violator of federal law," Stewart said.

The EEOC "misses the mark" in contending that a prevailing defendant could be awarded fees only if the complainant was motivated by bad faith, the justice wrote. Agreeing with the judge who rejected the garment firm's claim. Stewart said it is enough to show that the EEOC's complaint had not been "unreasonable or meritless" or "frivolous."