Three times in the last three years opponents of affirmative action programs have taken their complaints against "reverse discrimination" to the Supreme Court -- and three times, affirmative action has survived.

In 1978, the court said public colleges could legally consider race in deciding who they would admit. In 1979, it said emplyers and unions could take race into account in making training and promotion decisions. In 1980, it ruled that the government could use race as a criterion in handing out federal contracts.

But now there are signs that opponents of affirmative action may finally have won, not in court, but at the ballot box last November when Ronald Reagan was elected president. Reagan has regularly expressed his distaste for "reverse discrimination," and Wednesday his administration indicated that existing affirmative action rules will be among its early targets for "regulatory relief."

The administration's regulatory relief task force said it will focus specifically on regulations issued by the Labor Department's Office of Federal Contract Compliance Programs -- the agency that has required 275,000 federal contractors and subcontractors to establish hiring plans that include specific goals and timetables for employment of women and minority group members.

The contracts compliance office has a broad reach. Its dictates apply to every bank that holds government deposits, every utility that provides heat or light to a government building, every college that does government research, and to all the comapnies coast to coast that sell any product, from memo pads to missile guidance systems,to the federal government.

The agency's rules apply to all these contractors, whether or not they have a history of discrimination. And its remedy is tough: firms that do not establish the required hiring plans lose their government business.

Because it governs access to jobs -- always an issue of concern to minority and women's groups, never more so than in difficult times -- and because it began cracking down on some of the largest companies in the country during the Carter years, the contracts compliance office has been the focal point of the most heated recent arguments about the basic principle of affirmative action.

The administration has not made clear precisely what changes it might make in minority hiring requirements for federal contractors, but Wednesday's announcement suggested that Reagan will try to set "broad performance standards" to replace "tight specifications" in existing affirmative action rules.

Some minority groups said yesterday they fear that could mean a return to the earliest day of federal civll rights laws, when the regulations said, essentially, "do not discriminate," but did not require active efforts to overcome the legacy of previous racial discrimination.

President Johnson imposed the first affirmative action mandate in a 1965 executive order that established the contract compliance office. "Freedom is not enough," Johnson declared. "You do not take a person who for years has been hobbled by chains . . . bringing him up to the starting line of a race and then say 'You're free to compete' and justly believe that you have been completely fair."

From the first, the other side of this coin -- the recognition that discrimination for blacks or women could amount to discrimination against white men -- has made the affirmative action concept an issue of particularly emotional controversy. Among those expressing reservations about the principle has been Reagan, who says he is concerned about "some affrimative action programs becoming quota systems."

If the administration does cut back affirmative action in the contracts compliance program, it could presumably also turn its attention to similar regulations issued by two other federal offices: the Equal Employment Opportunity Commission, which has broad authority but less direct enforcement power than the contracts complaince office, and the Commerce Department, which controls "set-aside" rules assuring that a certain share of federal contracts go to minority firms.