The Federal Trade Commission won a round in its controversial three-year-old antitrust investigation of the automobile industry yesterday when a federal appeals court ruled that the auto industry could not fight the probe in court until it had exhausted FTC administrative appeals.

The 6th Circuit Court of Appeals in Cincinnati moved to dismiss a suit file by General Motors Corp., Chrysler Corp. and American Motors Corp. seeking to quash FTC subpoenas for company records. A lower court in Detroit last year had moved to quash the subpoenas at the request of the three automakers.

But the three-member court of appeals overturned that order, concluding instead that the manufacturers had to go back to the FTC to appeal the subpoenas before coming to federal court.Ford Motor Company did, in fact, take its appeal directly to the commission, which has already responded by sharply narrowing the scope of its Ford subpoena. Ford is, however, appealing the remainder of the subpoena to the courts.

The FTC investigation is seeking to determine whether or not the auto industry has been anticompetitive.

FTC attorney Dan Schwartz said the investigation, which had been hampered in its attempt to obtain evidence because of the lower court ruling, is nevertheless continuing. "We are obtaining information from both domestic and foreign auto manufacturers," he said.

In their suit, the automakers had called the FTC investigations "fishing expeditions." The FTC subpoenas had asked all four automakers for copies of all records bearing on automobile production and sales from 1946 through 1980.

General Motors, in a statement issued by vice president and general counsel Otis M. Smith, said yesterday's court setback "involves a purely procedural point. The automobile companies have claimed that the pending investigation by the FTC is unconstitutionally broad, unfocused and burdensome. The court of appeals expressly noted that it was not ruling on the merits of these claims."

Smith said GM "has not yet decided what further course of action it will pursue."