General Motors Corp. will roll out the last of its X cars in May, thus ending production of one of the most controversial car lines in the company's modern history.

The front-wheel-drive compacts entered the market as 1980 models in late 1979. They were hailed by the company and by automobile critics as an American answer to the fuel-efficient Japanese cars then invading the U.S. market. But the 1980 X cars soon became the target of a federal defects investigation that now has GM facing a potential $4 million in penalties in U.S. District Court here.

The cars included the Chevrolet Citation, Buick Skylark, Oldsmobile Omega and Pontiac Phoenix. The alleged defect in the 1980 models involves rear brakes that lock up in moderate-to-hard stops, causing possible loss of driver control.

GM dropped the Omega and Phoenix last year. It will discontinue production of the Citation (now called the Citation II) and the Skylark at the end of the 1985 model run in May.

Meanwhile, the National Highway Traffic Safety Administration has notified the auto maker that it is investigating potential rear-brake lockup problems in A models made for model years 1982 through 1984. The A cars include the Chevrolet Celebrity, Buick Century, Oldsmobile Cutlass Ciera and the Pontiac 6000.

An NHTSA spokesman yesterday emphasized that the A-car investigation is a preliminary matter that could lead to a recall.

GM repeatedly has denied that its early X cars were defective. The National Highway Traffic Safety Administration, through a suit filed by the Department of Justice, contends otherwise.

The government alleges that GM knowingly rolled out the 1980 X cars with defects and that the company then tried to cover up the alleged mistake. GM denies those charges, too, and predicts that it eventually will win the case that began last March 13.

The often-delayed, long-recessed, non-jury trial before Judge Thomas P. Jackson is scheduled to resume next week.

In statements yesterday, GM said, "The car has no design problems. It has been the victim, however, of adverse publicity, which is unfortunate because it is a fine car. The publicity undoubtedly has affected sales."

Ward's Auto World, a Detroit-based auto industry trade journal, gave the X car its 1985 "Loser of the Year" and "Wrath of Washington" awards, saying the line was "bludgeoned by relentless publicity over allegedly unsafe brakes on earlier models."

The Citation's recent sales fortunes indicate the effect. The company sold 93,000 Citations last year, but faced with declining demand, scheduled only 50,000 for production at its plant in Willo Run, Mich., for 1985.

However, the company said yesterday that an important factor in that phase-out is its production of "more modern cars that are doing exceptionally well" in the compact segment of the market.

The National Highway Traffic Safety Administration's investigation into GM's A cars is based on 448 complaints, involving an estimated 69 accidents and 23 injuries, the ageny spokesman said.

"NHTSA has merely indicated that it may open a formal investigation into the brake system of the A cars. . . . Only an extremely small number of complaints have been received on a population of nearly 2 million cars," a GM spokesman said. He said that "GM does not believe" that a formal investigation "is warranted."