American Motors Corp., hurt by a fall in sales of its subcompact Alliance and Encore cars, reported a first-quarter loss of $29 million yesterday.

The profit decline means a 29-cent reduction in earnings per share, compared with a gain of 3 cents on a net income of $5.1 million in the first quarter of 1984. The loss ended five successive quarters of modest earnings for the auto maker.

Auto industry analysts said yesterday that AMC's problems could continue for several quarters because of a slowdown in sales of small cars in the United States, particularly subcompacts.

Subcompact cars accounted for 16 percent of auto sales in the U.S. market in the first two months of this year, compared with 16.2 percent for the same period in 1984.

AMC sold 31,974 cars in the United States and Canada in the first quarter of 1985, versus 65,067 in the year-ago period, according to a company spokesman.

Subcompact sales have been fueled with incentives -- 8.8 percent finance rates, free automatic transmissions and other deals to encourage buyers in an era of relatively cheap gasoline.

Competition in the small-car arena also is getting intense. The Japanese government will allow its auto makers to ship 2.3 million passenger cars to the United States for fiscal 1985, a 24 percent increase over the official ceiling of 1.85 million cars for fiscal 1984.

General Motors Corp. and Toyota Motor Corp. are assembling 250,000 small cars annually at a joint-venture company, New United Motor Manufacturing Inc., in Fremont, Calif. Japansese auto makers Honda, Nissan and Mazda are assembling -- or planning to assemble -- their own small cars on American soil. In addition, Yugoslavia and Korea are preparing to ship thousands of "mini" cars to U.S. shores.

AMC is planning to beef up its product offerings with midsized cars. But they will not be ready until 1987, company officials say.

AMC sales for the quarter fell 15.1 percent to $919.4 million from $1.1 billion in the comparable period a year ago.