American Airlines, the nation's second largest air carrier, agreed yesterday to pay the Federal Aviation Administration a record $1.5 million in civil penalties for "maintenance-related violations" of safety rules.

The payment settles all issues resulting from a series of safety-related incidents on American flights and a subsequent special FAA investigation of American's maintenance practices.

Under the agreement, American does not admit the violations and the FAA agrees that the matter is concluded.

FAA Administrator Donald D. Engen said that American has taken prompt action to correct the deficiencies and that the airline is operating safely.

"I am pleased with American's cooperative attitude and aggressive corrective efforts," Engen said. "However, Congress has mandated and the public expects full accountability for an air carrier's failure to comply with the Federal Aviation Regulations. I intend to continue to apply the full force of statutory penalties to all violations."

Engen said the special inspection at American was one of a series of maintenance evaluations at major lines. Sources said, however, that none of the other inspections is apt to produce fines of this magnitude.

American, in a statement, praised the FAA's "audit" as one that "will serve the public, American and the airline industry well." It said that "the audit did reveal several deficiencies," but that "the fact that the FAA has assessed fines against the company is disappointing. Every American employe shares an unshakable commitment to safety . . . . "

Previously publicized incidents that prompted the special investigation include an engine falling off a Boeing 727 in flight and the use of plastic parts instead of required metal parts in the wings of McDonnell Douglas DC10 jetliners.

An FAA news release said the violations involved "maintenance performance, the improper postponement of maintenance and defects in procedures for monitoring performance and quality control."

FAA documents said that:

*A tire and wheel fell off a taxiing American DC10 because a retainer lock pin was not installed after a tire change. Two flights were made before the problem called attention to itself.

*Several flights on McDonnell Douglas MD80s were made with inoperable intercom systems connecting the cockpit and flight attendant stations, a potentially dangerous situation in an emergency.

*An MD80 was flown with an inoperative takeoff warning horn on one of the engine throttles. A takeoff warning horn sounds in the cockpit when the throttles are advanced but the plane's controls are not properly set for takeoff.

*The airline did not perform a mandatory inspection and repair of a lavatory waste access door.

The maximum penalty for one violation is $1,000, although Transportation Secretary Elizabeth Hanford Dole is seeking legislation to boost that to $10,000. Each time a plane takes off with an illegal maintenance deficiency is a violation, so a single discrepancy can result in a large fine if it occurs on more than one flight.

FAA letters to American proposed penalties totaling $417,000 for some violations, but the agency did not prepare formal letters on other violations. "We obviously feel he had enough there to warrant the settlement we reached," said E. Tazewell Ellett, FAA's chief counsel. A number of meetings were held between top FAA safety officials and American's management.

After the investigation started, American made several changes in its maintenance program, according to FAA and airline sources. The assistant vice president for aircraft maintenance has retired and the FAA reassigned its principal American maintenance inspector.

Aviation enforcement authorities said the settlement is the largest ever. In 1979, American paid a $500,000 settlement "under protest" for maintenance violations linked to the DC10 crash that killed 273 people in Chicago, and Braniff International Airways paid a $400,000 fine in January 1981 for various safety violations.