Every one of the 2,800 seats in Carnegie Hall was filled by 2:30 Saturday afternoon when the house lights dimmed. The 90-piece American Symphony Orchestra filled most of the stage and behind it, three rows deep, stood the 75 members of the Orpheon Chorale.

The great hall grew silent, and then a groundswell of applause erupted and blossomed as a tiny old woman in a long blue dress shuffled across the stage and, with considerable difficulty mounted the conductor's podium. She faced the audience, extended her frail arms and broke out into a huge grin. Then she turned around, sat in a straight-backed chair, raised her baton and went to work on Beethoven's sweeping "Missa Solemnis."

Charlotte Bergen left Walter Mitty in the dust ages ago. As they say, she had the means and the motives.

Four times a year, she spends about $40,000 of her own money to rent Carnegie Hall, the orchestra and an occasional chorale. And, at no charge, she throws open the doors of America's musical mecca to the public, which packs the place.

It all began three quarters of a century ago when, in her own words, "I sat beside my mother at my first concert wondering how it would feel to get up there and swing that stick."

For the last 10 years, "C. Bergen," as she calls herself, has been swinging it with the American Symphony Orchestra through her favorite music at Carnegie Hall.

"I took the ball and ran," she shrugged during rehearsals last week.

At 81, Charlotte Bergen is the envy of every armchair conductor who has ever waved an imaginary baton in an empty living room. She is also an inspiration to rich people who don't know how to spend their money well.

"Even if no one came," she confided, "I'd still do it."

That statement explains in part why New York is enchanted with her; she does what she damned well pleases. She is an amalgam of noblesse oblige and true grit.

The generous-patron-of-the-arts half of her belongs to the era of Peggy Guggenheim, and the other half remains firmly rooted on the tractor she drove for years on her 200-acre estate in New Jersey.

Music to Charlotte Bergen is like winning was to the late Vince Lombardi. "I don't think about anything but music," she conceded. "My feelings for music are long, beautiful lines of color."

She has been painting her musical pictures at various times in her life with the piano, the cello, the oboe and, of all things, the bugle. Her house is stocked with a huge organ, two pianos, two Stradivarius cellos and a mammoth classical record collection.

"If you don't want to talk about music, she doesn't want to talk to you," moaned one of the four women who take care of her.

Charlotte Bergen and her older brother, who died during World War I, grew up in a home full of music and money. Her mother, a member of the wealthy Gardiner clan, was an accomplished singer, and her father, the general counsel to the Public Service Company of New Jersey, began fostering her interest in music when she was 5.

Like many people born with a lot of money, she appears to take it for granted, but doesn't. She simply learned how to use it at an early age. Although she has seven people working for her in various capacities, "C. Bergen" wears no-nonsense shoes and sensible clothes. Her one vice appears to be an addiction to a pint of vanilla ice cream a day, often delivered to her estate in a taxi.

She started her own dairy during World War II and got up at 4 every morning, she says, to produce "the cleanest milk in northern New Jersey."

She grew wheat, too, and rode her tractor through the fields of Bernardsville, which is, home for other keepers of big money such as the widow of the late South African diamond and gold king, Charles Engelhardt.

But C. Bergen has eschewed the considerable social scene there all of her life. "I've always been a loner," she said. "I'm a hermit, but not a misanthrope. I just can't stand all of that tea party nonsense.

"I'm a homebody," she said. "I'm very constant there. Europe? I go to Europe in my music."

Charlotte Bergen played the cello for 35 years in a small room in the house that her father built, the place she says she will never leave. She began her conducting career with the choir at the Catholic church in Bernardsville, Our Lady of Perpetual Help. She loved the liturgical music and says she was heartbroken when Latin was dropped from the mass.

During the mid-'60s she conducted concerts in the Holy Trinity Episcopal Church on Manhattan's upper East Side and at Riverside Church, the gigantic structure built by John D. Rockfeller near Columbia University. Her audiences kept growing and she moved to Town Hall, which is close in size to Carnegie Hall, and finally to Carnegie itself a decade ago.

Most of her audiences consider her an old friend by now, and she says more than 6,000 of them have written to be put on her mailing list. She employs two people to handle the correspondence and to notify her fans when the next concert will be held.

"I've been coming to these for seven years," Gertrude Karp of Brooklyn said Saturday afternoon. "It's a beautiful idea, because we all can enjoy this. She's very generous."

"This is special because it's something elegant -- well thought out," added Mancy Radia, an aristocratic Romanian emigre. "I heard about it, and I came out of curiosity. I've been coming now for four years."

Some members of the American Symphony Orchestra are not as enamored of the concerts. They complain that she simply isn't up to the job professionally or physcially. But they regard her with genuine affection and appear willing to put up with her despite these drawbacks and her sharp temper. a

"Someone's making too much noise at the wrong time," she barked during Wednesday's rehearsal, oblivious to a number of orchestra members who rolled their eyes.

A Charlotte Bergen concert brings a different texture to Carnegie Hall. The grandeur of the place is tempered by a pronounced informality among her audience, made up mostly of elderly people speaking every language under the sun and a smattering of students in down parkas and boots. Their only common denominator is their appreciation of the music and the iron butterfly up on the conductor's podium.

For her part Charlotte Bergen is pleased. Who wouldn't be? "My only regret is that I should have done what I'm doing now sooner,; she said last week before hightailing it back to Bernardsville.