For nearly 18 years, Washington Ballet Director Mary Day virtually shanghaied the brothers of her female ballet students for the annual production of The Nutcracker. She had no dirth of applicants for pink tutus and toe shoes, but boys, essential for the first-act party scene, were hard to come by.

Today, however, the boys are starting to crowd the little girls. During a recent rehearsal, Day urged the girls forward, "so we don't see only boys." It was a novel situation for the Washington Ballet, which turned professional two years ago. The boys' willingness to dance may signal a change in the traditional prejudice against male ballet dancers.

The sugary confection known as The Nutcracker has become as much a holiday standard for children as the Messiah is for their parents. In productions such as the Washington Ballet's, the ingredients include flowers, mice, sugarplum fairies, magical Christmas trees and, in theory, little boys to chase little girls around the stage. Many directors have resorted to dressing little girls in knickers and hoping no one would notice.

Mary Day, surprisingly, no longer experiences shortages. In her current production, there are two casts of small boy dancers, some of them returnees from last year. Furthermore, more are students in her all-boy classical ballet class which began last year and has more than 20 members. Most small boys take classes with girls or take private lessons.

Ballet isn't as big as soccer with under-12 boys but it's growing in popularity.

Milling around the rehearsal hall in old jeans and black practice slippers the young male dancers, most in the 8-to-12 age range, showed the same boisterous enthusiasm as Little Leaguers an sidewalk stickball players.

In fact, only such enthusiasm could account for the sacrifices the children and their parents make for this art. The boys will perform in 11 or 12 performances, mostly at night. Edward Smith and Steven Tobin, the two 12-year-olds who alternate in the major role of Fritz, must commute an hour in and an hour back to all rehearsals, classes and performances.

Leesburg, Va., resident Edward Smith Sr., toted his portable television set to the rehearsal hall so he could catch the Redskins-Cowboys game while his son practiced stomping on the Nutcracker doll.

Smith said he was happy to accommodate to his son's "desire to dance." Young Edward, who started as an extra, or super, at the Kennedy Center, finds performers "nice people."

A center on his local football team, Edward started studying ballet because "he had no arches and the doctor thought it might help," his father said. His sister was a student of Day's so Edward was packed off to the ballet school as well. Now his sister has given up dancing for cheerleading; Edward's arches are improving, and he keeps on dancing.

The alternate Fritz, Steven Tobin of Columbia, Md., is a soccer goalie off the rehearsal stage. His dream is to become an actor and he is now trying to decide whether to launch his career in California or New York.

Steven talked about some of the problems a male ballet dancer faces in this country and said he regards them as an infringement on his basic rights.

"It's the same thing as women's lib, really. You shouldn't tell women what they should or should not be, and you shouldn't tell boys they can't dance," he said.

"Sometimes when a guy finds out that you dance he may not like you so much. Not too many people know that I do it, and I kind of keep quiet about it," Steven said. "But if someone asks, I try to explain that it makes me a better athlete and that it is very hard. But if they still don't understand, you're forced to ignore them."

Marcus Blum, 9, of Bethesda is convinced that the only way to convince the doubters is to have them try it. Marcus has made up his mind that he wants to be a ballet dancer and has already studied for four years. He was converted to ballet after attending performance by Dame Margot Fonteyn, where he saw male dancers.

"At first I thought dancing was really stupid, I know that somebody who has never taken ballet would think it was silly, but after you've tried it . . .," Marcus said.

Asked about his solution to people who put him down, muscular Edward Smith replied, "I beat 'em up."

Sighed the more slightly built Steven Tobin, "It's easy for him to say that, he's a football player."

Performances of The Nutcracker are scheduled from Dec. 15 through 31. Telephone 362-4644 for more information.