The Federal Communications Commission voted yesterday to relax substantially its regulation of smaller communications companies that compete with established telephone companies in providing long-distance service.

Approving a staff recommendation, the FCC voted to separate the long-distance-service competitors into "dominant" and "nondominant" categories. l

The Bell System, General Telephone & Telegraph Co. and the major independent telephone companies are considered "dominant" firms by the FCC under its ruling, and no deregulation was approved for them yesterday.

As estimated 24 smaller firms, including MCI Communications Corp. and Southern Pacific, which supply long-distance telephone services in competition with the telephone companies, do qualify for partial deregulation, the FCC concluded.

These "nondominant" firms will be allowed to raise or lower their rates without supplying the extensive economic data normally required by the FCC. Iin addition, they will be required to give the FCC only 14 days notice before making rate changes, rather than the 70 to 90 days normally required.

They will not be required to obtain the FCC's advance approval when expanding their services to new areas or adding service capacity. Once an application to do business has been approved, they merely must inform the FCC twice a year what areas they are serving, FCC officials explained.

The FCC asked its staff to prepare a formal proposal on a more controversial and farther-reaching deregulation policy that in effect would establish three categories for long-distance carriers based on a determination that the FCC is not obliged to regulate small firms that do not control a significant share of the market.

The major telephone firms, led by the Bell System, would continue to be regulated. The small, specialized firms such as MCI would be deregulated completely, and an intermediate group -- which presumably would include Western Union and RCA -- would be partially deregulated.