The nation's airlines, concerned over 11 recent hijacking attempts, asked President Reagan yesterday to "take prompt and energetic action" to seek the extradition of the most recent hijacker for prosecution to deter potential air pirates.

The Air Transport Association sent a telegram to Reagan and four Cabinet officers noting that a similar spate of hijackings in August and September of 1980 stopped immediately after Cuba extradited two of the hijackers to the United States for prosecution.

The State Department reported that as recently as July 22 it transmitted a note to authorities in Havana requesting either extradition of hijackers or prompt public announcement of their Cuban fate as a way of discouraging future air piracy.

In interviews yesterday, officials at both the State Department and the Federal Aviation Administration said diplomatic and security efforts were being pursued vigorously to halt the worst burst of hijackings since the fall of 1980.

Of the 11 attempts in the last 15 weeks, nine have been successful. Two have been thwarted by passengers who subdued would-be hijackers. Nine of the attempts have been by Cuban refugees, many of whom claim to be trying to return home to visit their families.

Distraught and homesick refugees have been accused of most of the hijackings. The State Department requested in its July 22 note that Cuba accept refugees who wish to return voluntarily. Cuba has refused consistently to readmit any of the refugees who fled in the 1980 Mariel boatlift.

"In refusing to permit the return of these persons," the department spokesman said, "the Cuban government is contributing to a problem which is in the interest of no one."

The Cuban government has not responded to those requests, but on July 20 Granma, the official government newspaper in Havana, published a "warning to future hijackers" saying that harsh penalities had been imposed on earlier hijackers and that the penalties would grow more severe if the crimes continued.

The FAA, meanwhile, has further tightened security and continues to emphasize that hijacking is not the way to try to get back to Cuba.

"The bottom line is that if the hijacker is unsuccessful he's going to face a U.S. prison, and if he's successful he's going to a Cuban prison," said FAA spokesman Jack Barker in Atlanta.

The FAA hopes by the end of the month to be broadcasting Spanish-language radio and television spots in New York and Miami to make that same point.

Barker said that at the same time more rigorous security measures are in force in both the Miami and San Juan airports, where most of the hijacked flights have originated or been destined.

The effort includes screening all boarding passengers and conducting more thorough searches of potential hijackers, installing more sensitive X-ray equipment in Miami and increasing the number of federal marshals riding shotgun on flights.

"The security system has been beefed up somewhat," Barker said. "But basically the security is good. There have been few breaches of security."

The most popular means of hijacking during the recent spurt has been to threaten to set fire to the plane with gasoline. A knife and several toy guns, as well as a signal flare pistol, have been used, but no real guns.

According to the FAA, there have been 116 successful hijackings of U.S. planes since 1961. During the same period, Barker said, 110 hijackings have been stopped or thwarted.