The Federal Communications Commission yesterday rejected an American Telephone & Telegraph Co. request for an interim increase in its maximum allowable rate of return.

The commission denied the increase pending action on a companion request for a permanent increase that would allow AT&T higher profits.

In denying the interim relief, however, the FCC ordered its staff to study AT&T's request for a permanent increase in its rate of return from the current range of 9.5 to 10 percent to as much as 11 or 12 percent.

The FCC said it will also study the issue of alleged excess profits that AT&T may have already earned, and the possibility of ordering as much as $160 million in refunds to telephone consumers.

The FCC decided in 1976 to allow AT&T to earn a rate of return of 9.5 percent on its interstate and foreign investment. At that time, the commission also said it would allow a "fudge factor" of 0.5 percent. That meant AT&T earnings could reach 10 percent, if it would mean increased productivity and efficiency.

But late last year AT&T revealed that it had earned as much as 10.22 percent in 1978, charging customers as much as $160 million more than allowed.

AT&T promptly filed a request for an increase in its rate of return, contending that it needed increased profits because inflation had forced up the cost of capital.

But at least one commissioner, Tyrone Brown, said no matter what is ultimately decided, the company should be made to return the earlier overcharges "because the funds rightfully belong to the customers."

In New York, AT&T said it was "naturally disappointed" in the FCC's refusal to grant an interim increase, but was confident that it could justify the need at upcoming FCC hearings.