Consumer groups and major retailers yesterday told the Carter administration that its plan for controlling consumer credit is unfair unwieldy and will do little to slow inflation.

Retailers told federal officials that the controls reach only about 5 percent of total consumer debt, may put stores in conflict with state usury ceilings and fail to take into account the seasonal nature of the retail business.

Consumer groups complained that the credit controls will wreak havoc with the monthly budget of millions of households. Several groups are preparing to take the Federal Reserve Board to court if it tries to suspend any consumer protection laws to allow lenders to change the terms of charge account or credit card bills.

In separate groups, about 20 retailers and about 50 consumer representatives were invited to the White House for briefings on the administration's latest inflation-fighting efforts.

The main tactic, announced a week ago by the Federal Reserve, is to try to put a cap on consumer credit. Any lender who increases outstanding credit balances about the levels of March 14 will have to deposit 15 percent of the increases in a non-interest bearing account with the Federal Reserve.

The retailer walked away with no assurance of any changes in the regulations but seemed generally satisfied with the meeting with Treasury Secretary William G. Miller, Federal Reserve Board Chairman Paul Volcker and members of the White House staff, according to Phillip Hawlely of Carter, Hawley Hale department stores.

"We felt good in the sense that they had a very open attitude without saying they could or would take steps," he said.

Many of the points the retailers raised had already been broached in a Wednesday meeting of corporate representatives and technicians from the Federal Reserve Board.

But beyond raising issues about the way the credit controls would be implemented, the retailers also suggested that the controls may reach only a small part of the problem.

Gas cards, retail credit cards and bank cards account for only 5 percent of consumer credit, with the rest being accounted for by mortgages, auto loans, home improvement loans and other secured debts, Hawley said at the meeitng. Of that 5 percent retail store credit only accounts for 2 percent according to Hawley.

In addition, said Hawley, as a portion of disopsable income retail credit had not increased in the last three years.

"We have not been a source of the big growth in credit. We're a very small piece and not a piece that's growing in relation to the total economy," he said.

On the other hand, the retailers told federal officials, the retail industry accounts for about 14 percent of all jobs.

When the retailers weren't talking about how small a piece of the problem they are, they were raising questions about how the controls would be implemented, including whether choosing one date as the key to how much credit can grow is fair.

"The thing that came up was just pointing out that retail tends to peak in October, November or December," said Hawley. "That raises the issue of whether the control level should be one number or should be related to typical peaks and valleys," he said.

Also discussed was whether it was fair for some purchases -- generally major home furnishing and appliance purchases -- to be covered when they were made through revolving credit but not under an installment plan. "The issue is should the controls relate to the type of merchandise or the type of credit account," said Hawley.

At the same time retailers were meeting at the White House over their concerns, oil company executives were questionting whether changing their credit card practices would conflict with Department of Energy regulations.

Representatives of Consumer Union, the Consumer Federation of America, the National Consumer Law Center and several state groups already have raised objections to the administration plan.

The chief complaint of consumer groups is that small borrowers are likely to bar the blunt of any limits on consumer credit. The family that has carefully budgeted its paychecks to handle monthly payments for several charge accounts will be in financial trouble if those payments are suddenly increased they complain.

Raising the minimum monthly payments on charge accounts is one of the steps lenders say they are considering to hold down the growth of credit.

Raising the payments on accounts that are already established is prohibited by the laws of several states. In other jurisdictions consumers must be given as much as six months notice of any changes in terms of accounts.

To enable the limits to be put into effect quickly, the Federal Reserve is considering using its emergency powers to temporarily suspend the state consumer protection laws and also part of the federal Truth in Lending law.

The Federal Reserve Board has made no decision to do that, but if it does, consumer groups are almost certain to challenge the action in court.