After five years of dramatic growth, Dulles International Airport saw a decrease in passengers in September, reflecting the decision of one airline to cut back its operations and a leveling of the growth rate at the airport, authorities reported yesterday.

The number of commercial airline passengers using Dulles dropped to 740,310 in September, down 8.9 percent compared to the same month of last year, according to Metropolitan Washington Airports Authority figures released yesterday.

The drop primarily reflected Continental Airlines' decision in July to eliminate more than a third of its flights at the airport, said Hugh Riddle Jr., deputy general manager of the airport authority, which operates Dulles and National airports.

Continental, which now operates 54 flights each weekday at Dulles compared with 86 earlier in the summer, said the move was part of an effort to better consolidate the operations it absorbed through recent mergers with New York Air, People Express and Frontier Airlines.

"We'll see ups and downs over the next months," in passenger levels at Dulles, Riddle said during a meeting yesterday of the authority's board of directors.

A flattening of the growth curve at Dulles would provide some relief from the spectacular increases in flights and passengers that have overwhelmed the parking, ticketing and baggage areas at the airport, which was once a white elephant.

The number of commercial flights in and out of the airport grew almost sixfold over the last five years, and more than doubled in 1986 alone. The last two years saw the greatest burst of growth as four airlines established hubs, where many connecting flights exchange passengers.

Even with the reduction in total passengers, Dulles reported a 5.2 percent increase in total airline takeoffs and landings in September compared with September 1986. In September, Dulles had almost 16,000 airline flights.

"The super growth experienced in the last 18 months is going to moderate," said Thomas G. Morr, president of the Washington Dulles Task Force, a private group that promotes the airport. "Our objective is to provide air service for a market that needs air service. We don't need growth for the sake of growth alone."

Several airlines have announced plans to add flights at Dulles during the winter, and United Airlines, the dominant carrier there since the Continental retreat, is expected to beef up its schedule in the spring after expanding its midfield terminal there.

Dulles continues to handle more flights than National, but National handles more passengers, according to the figures. National served 14.5 million passengers compared to Dulles's 10 million in the 12 months that ended Sept. 30.

In other action at the board meeting:Airport officials said that they are trying to find a way to move the Piedmont Airlines and USAir operations at National closer together because of their recently approved merger. The two airlines operate at almost opposite ends of the airport.

The board chose the law firms of Hunton & Williams of Richmond and Reynolds & Mundy of the District to advise the authority on how and when to issue bonds to finance an estimated $700 million in improvements at the two airports. The board also chose Aviation Planning Associates Inc. of Cincinnati, Abacus Technology Corp. of the District and Price Waterhouse to provide additional planning help.