The agreement between two automotive giants, General Motors Corp. and Toyota Motor Co. to produce a subcompact car together in California has raised unique, difficult antitrust questions for the Federal Trade Commission and the rest of the American auto industry.

The key issue is whether GM--with Toyota's aid--would be able to increase its dominant position in the American car market.

A legal hot potato described by antitrust experts as "a very close call," this and related questions rest with the FTC, whose staff has been studying the potential impact of the venture since it first was rumored in the fall.

GM, the world's No. 1 automaker, and Toyota, No. 3, clearly fashioned their agreement to avoid U.S. antitrust problems. The joint announcement Monday in Tokyo and Detroit emphasized the limited goal of producing a small car of Japanese design in the United States. The deal will last no longer than 12 years and involves no other forms of cooperation between GM and Toyota.

Despite the two companies' domination of the world car market, Toyota Chairman Eiji Toyoda said in Tokyo that he believes the joint venture falls "within the scope of approval by the FTC."

Lee Iacocca, chairman and chief executive officer of Chrysler Corp., disagreed, calling "the GM-Toyota deal . . . clearly bad."

It puts world markets within the dominating grasp of two companies that together already control 25 percent of the world's auto sales," continued the outspoken Iacocca.

GM now controls about 45 percent of the U.S. market. There are fears within the industry that, because of its joint venture with Toyota and increasing protectionism in the U.S. market, GM will end up controlling as much as 60 percent of the domestic market--further weakening Ford Motor Co. and Chrysler.

Ford, America's second largest auto manufacturer, has maintained a public silence while making little secret in private of its opposition. According to auto industry sources here, Ford's attorneys have filed a legal brief attacking the GM-Toyota agreement.

At risk if the GM-Toyota car becomes a sales success in the American subcompact market is Ford's Escort, which will be four years old in design when the new entry first rolls off the assembly lines in late 1984, and Chrysler's Omni, which is even older.

The FTC has been badly split on ideological grounds over the past two years. However, in this case, both the commissioners and staff appear to be approaching the issue strictly on its legal merits. There are no indications yet which way they are leaning. Commission Chairman James C. Miller III has excused himself from cases involving GM because the giant automaker is a former client.

The legal issues are complex. FTC Bureau of Competition Chief Thomas Campbell said basically they revolve around whether the new GM-Toyota car would reduce current or potential competition in the subcompact market.

A "yes" answer on either count would run afoul of the FTC's antitrust policies.

GM currently offers the Chevette, first introduced in 1976 and little changed since, in the subcompact field, while Toyota produces a Corolla subcompact. Presumably to blunt possible antitrust objections, GM officals have insisted they will continue production of the Chevette, while stressing that the new car will be "distinct and separate"--not a version of the Corolla.

A fleet of lawyers will be waiting alongside the auto executives, sales staff and customers, to see the design of the new car.