Officials of Washington's Arena Stage had a problem, and just one look around the audience on a typical night was all it took to see it.

The theater subscribers were getting older -- median age 52, according to a survey five years ago -- and no younger ones were replacing them. The theater also needed to attract a wider audience to bring in more revenue for more ambitious productions.

Officials embarked on a campaign. Younger people do not like to buy tickets in advance by mail, so the box office and ad campaign was revamped to emphasize telephone sales of single tickets to impulse buyers as late as the day of performance.

It worked: 90 percent of the Arena's single-play tickets are sold by telephone, a reversal of past practice.

Like all nonprofit organizations, the nation's museums, theaters and other cultural institutions are squeezed by tighter funds and rising costs. Borrowing techniques from business, they are expanding their professionally trained staff, running sophisticated ticket-selling and fund-raising campaigns, and operating high-priced boutique-like shops.

"The only thing that separates the aggressive, entrepreneurial nonprofit institution from others is that the ones that aren't so aggressive haven't learned how to be yet," said Peter C. Marzio, director of Houston's art museum and the former director of the Corcoran Gallery of Art here.

Marzio is enthusiastic about the trend, saying it helps make more art possible. But critics argue that the changes run the risk of watering down good art or sacrificing the offbeat masterpiece in the quest for financial stability. Marzio concedes that most museum officials are uneasy: "There's a feeling that commercialization will invade the esthetic purpose."

In fact, at least three major groups representing arts organizations warned recently that artistic adventurousness is being dampened by the emphasis on bringing in money. And the National Endowment for the Arts has commissioned a study of the "artistic deficit," the industry's name for the compromise of art to commerce.

In a widely noticed speech last year, Philippe de Montebello, director of the Metropolitan Museum of Art in New York, decried "the growing trend to convert art museums into hyperactive activity centers and curators into full-time artifact arrangers in order to generate public interest and lure visitors which improves receipts as much as to provide pleasure and enlightenment." He warned that "preoccupation with finances may unwittingly lead museums away from the very standards of quality that the funds are meant to uphold." A Museum McDonald's

One example of the new spirit is the California Museum of Science and Industry in downtown Los Angeles. It is among three in the country in which the museum restaurant is a McDonald's, only this one includes a display on how to run a franchise. The museum gets a cut of the profits. "It's a great gimmick for them," said Don M. Muchmore, the museum director, "and for us it's overwhelmingly successful."

Muchmore, a savings and loan executive who returned in 1982 to direct the museum he had managed years earlier, wooed businesses and others into donating $48 million for four new buildings, an outdoor air and space garden, and a theater, with corporate names mentioned prominently throughout.

The museum, which has two gift shops, also draws crowds by fingerprinting children on weekends and staging ethnic festivals, complete with ethnic food. Museum attendance is up 70 percent in three years.

Muchmore, who describes his philosophy as a combination of Aristotle and P.T. Barnum, acknowledges he is not without controversy: "I have a lot of museum directors look down their noses at me." One reason, he said, is, "I don't stuff animals."

Other examples:

*The Pittsburgh Symphony hired a staff of 30 telephone salespeople, began a subscription drive that mailed 14 different letters, each aimed at a different demographic group of prospects, and ran an advertising campaign describing its concerts as a cure for life's stress and worries. Subscription sales are up this year for the first time since 1979, according to marketing director Louis G. Spisto, who said new sales alone brought in $150,000 more than was spent on the campaign.

*New York's public television station WNET recently put on a breakfast to promote its recent venture into "enhanced underwriting" -- the sale of advertising spots to buyers who are not underwriting a particular program. The station hopes to make $1.2 million a year. Station officials, sensitive to criticism, say the ads may not promote a product, and are allowed by the station's charter and the government. WETA in Washington has no such plans, at the moment.

*The Kennedy Center for the Performing Arts is among many organizations that rent out their facilities to make money. Four hundred spaces in the underground garage are rented to monthly parkers for daytime use for a competitive $80, and 800 to 900 more are available on a day-by-day basis to paying customers such as tourists. Some patrons complain that the garage is full when they arrive for matinee performances. "We're sorry for the inconvenience," Kennedy Center spokesman Laura Longley said, "yet at the same time we have a responsibility to generate income."

One aspect of the trend is greater reliance on professional managers and consultants. More are being trained: There are 27 university-based art management programs now, compared with three in 1973. Another clue to the new emphasis on business: theater administrative staff members increased 30 percent in five years, according to the Theater Communications Group, a nonprofit theater association.

For performing arts organizations, there is more attention to sophisticated telephone fund raising and subscription sales. Costs of raising funds for the nation's opera companies, for example, nearly doubled -- from $4.3 million to $8.4 million -- between 1980-81 and 1982-83, according to Opera America, a trade group.

At Arena Stage, managing director Tom Fichandler used to raise money by himself. Now, he has four assistants, and income has more than doubled in five years -- from nearly $3.2 million in fiscal 1980 to $6.5 million in fiscal 1985. A slightly bigger share of the budget now goes to administration.

At museums, officials are expanding their shops and mail order catalogues. More are selling souvenirs and booking blockbuster exhibits that bring in crowds such as the treasures from British country houses exhibit that opened at the National Gallery of Art this month. New York's Metropolitan Museum of Art even has a shop at Macy's.

For the arts, money pressures stem partly from dashed hopes. The arts became accustomed to giant growth in the early 1970s, fueled by a National Endowment for the Arts program budget that soared 700 percent between 1970 and 1974, to nearly $61 million. But in the Reagan era, from 1980 to 1984, NEA's program funds grew less than 25 percent, from $97 million to nearly $119 million.

The spectacular growth of the arts a decade ago means intense competition in slower times for audiences and money. "You pick up the newspaper and you hardly have time to figure out all the things you want to see," said Donald Schoenbaum, managing director of the Guthrie Theatre in Minneapolis.

Some changes are prompted by the natural evolution of arts organizations. Once run on enthusiasm, borrowed space and spartan budgets, many now are becoming like well-established businesses with bigger financial needs. Often, a charismatic artist-founder is stepping aside -- or being pushed -- in favor of a board of directors more concerned about preserving a growing organization.

The nation's newest patron of the arts, corporations, are pushing hard-nosed business practices. And artists "are more demanding of creature comforts," which further strains the budget, said the artistic director of one modern dance company.

The money crisis in the arts is not new: Artists always have had to balance their desire for creative freedom with their need for cash. But William J. Baumol, the nation's reigning arts economist who teaches at New York University and Princeton, said the current money shortage is deeper. It is forcing organizations "to react in a much more visible manner," he said. Negative Effects Cited

According to the Theater Communications Group, money concerns are prompting theaters to discourage extra rehearsal time, research and new plays.

"They used to take chances on plays because they felt it was important or it made a statement," said Peter Zeisler, TCG director. Now, he said, "They're tending to use the same criteria that a commercial producer uses: What is going to sell?"

The example some theater people cite is the Guthrie Theater in Minneapolis, where the board of directors ordered the artistic staff to put on a production last spring of "Anything Goes," a 1930s Cole Porter musical somewhat out of tune with the institution's classical repertory. "There are those works that you do to allow you to do those works that are less acceptable," managing director Schoenbaum said. He said the Guthrie must be particularly wary of a potential flop because of its huge size; at 1,441 seats it is double the capacity of the Arena Stage and most other nonprofit theaters.

At the Pittsburgh Symphony, "The programming this year is geared more to the community than in the past . . . a slight shift toward the grand masterpieces," marketing director Spisto said. "Our main objective during the next couple of years is to build our audience base. Then we need to develop it from the artistic standpoint, and lead."

Ballet companies these days are relying more on the popular "story ballets" such as "Romeo and Juliet" or "Sleeping Beauty," said Donald A. Moore, executive director of Dance/USA, a trade group with 58 members, but it may be an artistic trend.

One well-placed theater official told of seeing a memorandum from the board to the artistic director at a major American theater that pointed to a marketing concern: There were too many foreign names in the repertory, it said. Couldn't something be done?

The American Association of Museums, in a report last year by a panel that included some of the leading names in museum life, warned that special blockbuster exhibits are diverting resources from the important tasks of collections' care and maintenance.

"In competition with the more visible public programs and popular special exhibitions, which offer immediate, tangible rewards to the museum and for which funding is more often available, the less glamorous, behind-the-scenes activities can too easily be pushed aside," warned the report, "Museums for a New Century."

Gregory Kandel, a Connecticut-based arts headhunter, said more institutional clients these days want artistic directors who are "flexible" and agreeable to compromise -- theater directors to do lighter musicals, ballet masters to stage the Christmastime staple, "The Nutcracker." Trend Called Temporary

The trend's defenders say the shift toward more popular programming is necessary, and temporary. "When organizations are struggling for financial stability, they are forced to take conservative postures," said Jonathan Katz, executive director of the National Association of State Arts Agencies and former director of an arts management program. "It doesn't help an artistic director to be unemployed."

Some say it is all to the good that institutions once run by and for the elite are responding more to popular demand. Futurist Joseph Coates, who thinks the arts need to broaden their public appeal, asks:

"When was the last time you saw a 12-year-old buy Mozart on tape?"

R. McNeil Lowry, former head of arts programs for the Ford Foundation and now a consultant, said cultural groups should not allow others to dictate their artistic agendas. "When the outreach program or special exhibit initiates with the staff as a part of their own conception of the mission, fine. When they do these things only to get money . . . they risk corrupting their institution into a nameless channel for funds."

Lowry also wonders about the cultural tone set by museum gift shops that increasingly look like high-class department stores. "Post cards, okay," Lowry said. "But T-shirts and terra cotta -- it's kitsch, and people take it home and say it's art."