Like an atomic scientist searching for the most elementary constituents of matter, Laura Dean set out about eight years ago to discover the basic particles of the dance experience.

It's this fundamentalism that has made her one of the most interesting choreographers of the modern dance scene. At the same time, part of the fascination has been watching her evolve toward more and more elaborate forms - from atoms, so to speak, to molecules to complex compounds.

Four years ago in a memorable concert at the Renwick Gallery, Dean and her small company presented among other things a work called "Spinning Dance" which consisted of just that and nothing else - dancers fixed in place, whirling about their own central axes, in silence.

Now, at American University's Clendenen Theater through Sunday, one can see how much ground she had covered in a new piece called "music and dance for nine," commissioned for and performed by an AU dance ensemble, with music by Dean and AU student David Koblitz.

In what she now smilingly refers to as her "early piece had one step, maybe two. The vocabulary opened up considerably in '75 and '76, to about 15 steps. It's a slow business for me. I still haven't found the richness that I'd be satisfied with, but I feel I have my own style, and I know exactly which direction I want to go in."

The basic ingredients of that style have remained the same throughout the evolution - simple walking, stamping steps; repetitive pulzation; geometric patterns. The ritualized repetitions of a Dean dance give it a mesmerizing effect and the same trait links her work to other artists - Meredith Monk, Robert Wilson, Kei Takei - in the vanguard of recent choreography and theatrical activity.

"I'm not a 'process' choreographer, though," Dean says. "I can't just toss out movement. I make dances like an architect makes a building - I have a need to see the whole structure at one time. The student pieces I do at universities, like this new one for AU, I can put together very quickly because there are really only introductions to my style. But the repertory pieces for my own company take tremendous amounts of concentration and time, months in fact. Right now I'm allowing myself a whole year to prepare a new work that will be premiered at the Brooklyn Academy of Music in July of '79."

As her choreography has become more intricate, other elements of a Dean performance have broadened their scale as well."Spinning Dance" had the performers dressed uniformly in white. In "Song," which Dean and her troupe performed at AU two years ago, the costumes were in shades of plum and rose.The new opus currently on display at AU calls for satin crepe garments for the eight dancers, in a bright variety of hues.

In recent years, Dean has composed most of the music for her pieces herself, using drones and repetitive phrases in the manner of her mentor, Steve Reich. The dancers also contribute to the soundscape, humming and singing as they move. The new AU piece calls for marimba, drums and voice.

Dean's underlying impulses remain purist, though. "I'm just beginning to feel what color is all about in a work, now. But white is my favorite color - there'll always be a white piece somewhere in my repertory."