As frequent flier mileage award programs have grown in popularity, they have produced a new business.

You can find it in the classifieds between the tickets to Bruce Springsteen and the hockey game, or on the back pages of Frequent Flyer magazine -- the advertisements for the brokers, willing to help those with more mileage than they can consume by matching them with buyers.

The only hitch is, according to the airlines, it's illegal.

Although many airline frequent flier awards allow the recipient to get a certificate for travel or to buy a ticket made out in someone else's name, most airlines prohibit selling the awards.

Enforcing the ban is another matter, however.

On the one hand, airlines don't want to give away seats to passengers who might otherwise pay. On the other hand, they want to keep their bonus programs sufficiently flexible to be attractive. Frequent flier programs are an extremely important marketing tool for airlines. The programs are designed to attract passengers who fly full-fare.

For instance, said Trans World Airlines director of business marketing James Smith, if a customer with 200,000 miles in his account appears to have sold a ticket in violation of TWA's rules, the airline wants to check the situation out carefully before it shuts down that account.

"The guy's spent some bucks on you," he said. "You don't want to make your best customers mad. You do walk a fine line.

Airlines are getting increasingly aggressive, though, about trying to persuade brokers that their business is illegal.

Northwest Airlines runs an ad that appears next to the brokers' advertisements in Frequent Flyer magazine, warning that "Northwest and Republic Airlines Frequent Flyer Awards are void if sold." Last month the airline filed suit against two Minnesota brokerages, World Connections and The Minnesota Traveler Corp., charging that the businesses operated in a "black market" for frequent flier certificates.

"The activities of these coupon brokers have cost Northwest millions of dollars in lost revenue," complained the airline, which also said that the brokerage business amounted to "a pattern of racketeering."

The brokerages deny the charge, portraying Northwest as an airline trying to change the rules in the middle of the game. "In the case of Northwest, they have imposed restrictions against buying and selling of certificates rather recently," said Peter Johnson, an attorney who represents the firms that are the target of the suit. Johnson said the airlines' restrictions took effect Nov. 13, 1986, and that the changes have not yet been filed with the Department of Transportation.

Johnson said the suit also amounts to "an attempt to shut down the brokerage business without directly confronting their frequent fliers."

Mark Shore, president of Go In Class Inc., an unrelated brokerage that operates in Chicago and New Jersey, said the airlines' efforts "are geared at getting the passengers buying the awards, as opposed to the frequent flier." Brokers deal with a very small percentage of airline customers, he said -- the most frequent fliers.

Their customers "are on the airline every day of the week, and the last thing they want is a free trip on the airline," he said. Those fliers, however, will go out of their way to fly a particular airline to accumulate bonuses to sell, he said. That kind of customer loyalty is valuable to the airline, he said.

The most valuable awards, for travel to offshore destinations, have been the hot tickets. Fares to foreign destinations are generally set by bilateral negotiations and are not discounted, so savings are biggest for passengers who can find a low-cost ticket for those flights. By the same token, when those passengers produce no revenue, the loss to the airline is painful.

"Especially when you're talking about first-class travel, it's an awful lot of money," said Susan Baer, manager of business programs for TWA. A first-class ticket to London might usually cost about $2,000, she said.

"Most of the airlines have started putting more fences around the awards that are more desirable," said Beverly Wilson, marketing manager for business programs for Eastern Airlines. Some awards are only available to the flier who earned them and a family member traveling with the flier. Others may be transferred only to another member of the family -- a term that is sufficiently broadly defined at some airlines to include first cousins and their spouses and in-laws.

Some awards are now unavailable or harder to earn for travel during periods of heavy traffic. A few airlines are also limiting the number of seats on each flight that can be booked through frequent flier awards.

Airlines say they believe the heyday of brokering is in the past, and brokers say their business is being made more difficult by the airlines' actions.

Airlines say they want the awards to be used so that large liabilities will not accumulate on their books, but brokers say the airlines expected to reap the benefits of customer loyalty without having to pay the freight in redeemed awards.

Airlines did not anticipate how many seats would be booked as a result of the bonus programs, said one broker who spoke on the condition that neither he nor his company would be named.

Once the airlines realized how well the programs were being utilized, they started trying to make them harder to take advantage of, he said.

"The airlines are targeting us," he said. "They used to like us, but they don't like us anymore."