The Carter administration has changed the way the government deals with institutions suspected of violating federal civil rights laws, and complaining civil rights groups are calling it a "retreat."

The change involves institutions that face pending charges of failure to desegregate. The question is whether to grant such institutions money to start up new programs.

In the past the government has not. But an interview over the weekend president Carter said "deferral" of federal funds from such institutions would be more selective than in the past.

"In other words, there won't be any massive withholding of federal funds that will hurt all the students in the university system or even a single college," Carter told a group of editors.

Two weeks ago, the Department of Health, Education and Welfare notified Virginia and Georgia tha they had submitted inadequate college desegregation plans. HEW also rejected North Carolina's desegregation plan for its four-year colleges.

HEW Secretary Joseph A. Califano Jr. said he would give the states another 45 days to negotiate acceptable desegregation plans with his department. If the three states still had not produced satisfactory plans by that time, HEW would start administrative proceedings against them that eventually could result in the yearly loss of millions in federal education aid.

In the past, according to Califano, HEW has blocked all future applications for federal funds from affected isntitutions once administrative proceedings have begun.

In the three pending cases, however, Califano said, "I will hold up approval of new applications for federal funds from such institutions only if those new funds would contribute to continuing segregation in those systems of higher education. In no case will student financial aid be affected . . . This fund deferral technique tailors deferrals more carefully to the underlying civil rights problem we see to remedy."

"That is a civil rights retreat," said Joseph Rauh, an attorney for the NAACP Legal Defense Fund, which sued the states to desegregate their systems of higher education.

"Califano has not complied with Title VI of the Civil Rights Act," Rauh said. "I think the way to enforce the law is to enforce the law . . . It's time to stop playing patty-cake with people who violate the law."

Rauh said the NAACP Legal Defense Fund has asked the U.S. District Court for the District Columbia to order Califano within 30 days to suspend all aid to traditionally white institutions of public higher education in Virginia, Georgia and North Carolina until the three states submit an acceptable plan for higher education.

Last April, a federal judge ordered six states to submit acceptable college desegregation plans to HEW by Feb. 3. Plans for the three other states - Arkansas, Oklohoma and Florida - were accepted by HEW.

In his interview with a group of broadcast and newspaper executives from outside Washington, Carter said, "I don't deplore nor do I disagree with the action HEW has taken."