Business is booming at the Smithsonian Institution, but not just in the museums' marble exhibit halls. The 13 gift shops and mail-order catalogue are ringing up $27 million in annual sales, twice the receipts of five years ago. The hottest item? Freeze-dried ice cream, the kind the astronauts took into space. Customers spend more than $375,000 a year on it.

The Pittsburgh Symphony courted the singles market this year by selling a six-concert series with an added attraction: a reception after each performance, where subscribers can listen to a jazz trio and eye each other around the cash bar. The series sold out and helped to reverse six seasons of declining subscription sales.

At the Washington Hospital Center, affluent patients can pay up to $560 a day for deluxe rooms that feature extra nursing care, gourmet food and a pullout couch for overnight visitors. The hospital laundry earns extra money by washing linens for other institutions. And a commercial subsidiary of the hospital's parent corporation runs parking garages and sells construction management services.

These and other nonprofit organizations, faced with increasing financial pressures, have been changing the way they do business. Discarding the sometimes relaxed, often idealistic practices of the past, they have been operating more in the modern management style of their profit-making counterparts. The result is greater solvency, and some troubling questions.

In many cases, nonprofit groups are selling new products and services in hopes of improving their bottom line. Others have tried to cut costs. Universities, for example, have dropped unpopular courses, and theater companies are staging less costly productions.

"The sleepy days when things could just go along and somebody took care of you are over," Ward Chamberlin, president of the corporation that runs the Arlington-based WETA-TV and WETA-FM broadcasting stations, said of the nonprofit sector. "It's now a highly competitive business, whether you're talking about museums or public television or libraries or zoos."

One organization trying to expand its revenue base is Planned Parenthood Federation of America, which began marketing "Planned Parenthood" condoms two years ago.

A cost-cutting move at the Cleveland Playhouse was made a year ago, when the new managing director, Albert Milano, closed one of the theater's two main entrances. The move raised eyebrows but enabled him to drop two guards from the payroll and reduce energy costs. "We saved $1,000 a week," he said, adding that he put the money into the artistic budget. Critics Raise Questions

Defenders say the changes have saved some institutions from bankruptcy and enabled others to channel more money directly into programs. But questions have been raised by a variety of critics. Some ask whether the developments could diminish the institutions' commitment to culture, academic freedom and social responsibility. And profit-making rivals complain of unfair competition from nonprofit institutions and the tax exemptions they enjoy.

The nonprofit sector is extensive: A study conducted by the Urban Institute reported that there were 124,000 active nonprofit organizations in 1982, two-thirds of them formed since 1960. Excluding religious and trade associations, nonprofit organizations spent $131 billion that year and employed 6.5 million people, accounting for about 5 percent of the domestic economy.

The changes in these organizations stem from many causes, primarily economic. The competition for funds spurred by the growth in the number of nonprofit institutions was heightened by Reagan-era cutbacks in government support. At the same time, philanthropic foundations have turned away from subsidizing day-to-day operations of nonprofit groups, encouraging them to aim for self-sufficiency.

Foundations, government agencies and corporate donors, the newest source of nonprofit funding, are demanding more accountability for the money they provide. Moreover, trustees of nonprofit organizations are mindful of recent court rulings holding them personally liable for their organization's business practices, according to Harvard business professor Thomas J.C. Raymond.

President Reagan's call to capitalism also has made its mark. "There's just a spirit of entrepreneurialism in the country right now," said Laura K. Landy, director of the Institute for Not-for-Profit Entrepreneurship at New York University.

WETA, the local public television and radio station, is augmenting its traditional membership drives with a variety of entrepreneurial tactics: renting use of its radio tower to taxicab companies, sponsoring a musical tour of Austria and Switzerland, and staging intercity teleconferences in the station studios. Following Corporate Model

The move toward better business practices first surfaced in the 1970s as nonprofit organizations started hiring trained managers with master's degrees in business administration. These managers first concentrated on cutting costs internally, then began searching outside for new ventures to bring in money.

Nonprofit organizations also borrowed other components of the corporate model: long-term and "strategic" planning with the aim of finding a niche in the market, more sophisticated cash management, and reliance on computers to track donors and file personnel records.

Washington Cathedral, facing a $10.5 million debt in 1978, brought in the Rev. Provost Charles A. Perry, an Episcopal priest with a master's degree in public administration. He, in turn, hired a management expert. Over the next five years, helped by fund-raising consultants, the cathedral erased the debt and raised enough money to resume construction of its unfinished facade.

The cathedral's fund-raisers have been so successful -- the $1 million annual fund is quadruple what it was nine years ago -- that they have won several awards from fund-raising trade groups. Also, the cathedral earns $200,000 a year from its gift shop, and it is experimenting with catalogue sales of products fashioned from original artwork in the church.

According to supporters, the recent developments enhance programs. At the Smithsonian, for example, officials say all gift shop items are related to museum exhibits and the profits from their sale enable the museums to sponsor activities not included in the federally funded budget.

But doubters say that these institutions are shifting attention from the programmatic questions -- what play to produce, what courses to schedule, what services to offer -- to another question: What will it cost?

"It's got some bad aspects to it," Chamberlin said of the trend. "Many of our enterprises are looking more commercial than we would like to have them, and some of the services are not half as good as they used to be."

This type of criticism was leveled at a recent move to improve cost-efficiency at three hospitals owned by Prince George's County. New managers fired 615 employes and ordered that patients who could not prove their ability to pay be refused some nonemergency treatment.

Some county residents and officials charged that this was a retreat from the traditional ethic of service first, money later. "If people are sick, they deserve care. That's why we had county hospitals in the first place," said County Council member Hilda Pemberton. Tax Exemptions Queried

Developments in modern revenue-raising can prompt questions as they change the image of traditional institutions. The downtown Washington YMCA, for example, has reversed its financial fortunes in part by offering an $800-a-year package that includes private lockers, massage and state-of-the-art Nautilus equipment.

A Y official, sensitive to criticism that the fancy facilities conflict with the organization's historic mission of social service, pointed out that the revenue -- $1.98 million in fiscal 1985 -- has kept the Y alive and helped produce a $55,000 surplus that went to programs for disadvantaged youth. "We don't have designer towels," he added.

There also is the question of the nonprofit organizations' tax exemptions. These organizations do not pay taxes on their income as long as the activity that generates it is related to their mission. That is a broad enough definition to include the Smithsonian's sale of a $44 "exclusive adaptation of a 1925 original sled." Nonprofit groups get reduced bulk postal rates and, in many states and cities, exemptions from sales and property taxes.

One reason behind the tax exemption was the notion that nonprofit organizations would provide a service that the commercial market would not supply. But critics, including some in the U.S. Small Business Administration, are asking whether computer software makers, engineering consulting operations or nursing homes run by nonprofit groups are entitled to special treatment.

"The traditional rationales for exempting nonprofits from federal income taxation do not withstand close scrutiny, particularly as applied to the modern-day 'commercial' nonprofits that offer goods and services in direct competition with for-profit firms," concludes a report by the SBA's Office of Advocacy.

Joseph F. O'Neill, chairman of the two-year-old private sector Business Coalition for Fair Competition, cites as one example of unfair competition a commercial laboratory that lost a customer to a nonprofit engineering school, which, because of tax and other advantages, could charge one-fifth of the commercial price for the same work.

"You've got the same market and the same type of activity, but two sets of rules -- the nonprofits playing by one set and the for-profits by another set," O'Neill said.

In some cases, though, nonprofit institutions are competing under the same rules. Some have established for-profit subsidiaries or operations that are subject to taxation in order to perform such traditional business as real estate development.

One of the earliest nonprofit institutions to become involved in entrepreneurial ventures was George Washington University. It developed and now leases several blocks of prime downtown Washington real estate, a venture that has helped boost the university endowment from $8 million to $170 million in 20 years. Last year the university paid $2.8 million in real estate taxes on five commercial buildings with net rentable space of 1.7 million square feet.

"You're beginning to see a blend of the profit and nonprofit sectors in America," said Peter C. Marzio, director of Houston's art museum, and formerly of the Corcoran Gallery here. "You're going to see a hybrid institution come into existence that is going to be a blend of the profit and the nonprofit." Cutting Back for Cash

While many organizations can claim greater solvency, there have been critical adjustments in the programs of many nonprofit groups.

The Arena Stage here has begun booking shows with small casts and minimal sets, such as The Flying Karamazov Brothers, a juggling act, or entertainer Stephen Wade's "Banjo Dancing," to subsidize the more traditional offerings that are more expensive to produce.

"The Flying Karamazov Brothers allows us to continue our mission . . . as it expands our audience," explained marketing director Richard Bryant. "It's fueled our growth."

Two years ago, Georgetown University eliminated eight graduate programs considered weakest and least able to attract students. Dropped were programs in Russian, Chinese, Japanese and Portuguese.

The cuts reflect economies that, despite their efficiency, will mean less variety in program content, said business professor Eric van Merkensteijn of the University of Pennsylvania.

"What you will lose are those things that occur at the margin for the fringes," he said. "The price is going to be a little less diversity."

NEXT: The education business