In art, the experimental or avant garde is generally either rejected outright or gradually absorbed into a more widely accepted category - not "mainstream," necessarily, or winning mass popularity, just no longer seeming particularly out of place.

In jazz, the constant striving for new musical expression results in many artistic efforts that appeal to a very limited audience - which is a problem, for a player must be heard to make an impact. And no record company can afford to continually back a product that will never emerge outside of its own closed circle of admirers.

So it's both refreshing and heartening to note that several of the more experimental jazz artists, who until recently were known only among aficionados and were rarely seen outside very small clubs and jazz lofts, are becoming full-fledged members of the popular music community. What's more, the transition has not affected their musical identities.

THE Art Ensemble of Chicago, who appeared at the Bayou this week, are a spinoff from Chicago's Association for the Advancement of Creative Music - a philosophically oriented group of musicians that also includes Anthony Braxton, Muhal Richard Abrams and Air's Steve McCall. Until their recent ECM release, "Nice Guys," the Art Ensemble of Chicago was generally grouped with other loft players whose surface posturings included indecipherable jargon and music that was hard on the ears.

"Nice Guys" is not particularly easy on the ears, but it is a rich, complex album that beautifully illustrates the maturing of the "new jazz." Throughout the record, there are hints of Miles Davis and John Coltrane and, just to show that this music also has a sense of humor, there is a Caribbean interlude that sounds like a tongue-in-cheek tribute to Sonny Rollins' "St. Thomas." Bassist Malachi Favors, who has played with traditional players like Dizzy Gillespie and Freddie Hubbard, is especially effective in his counterpoint with percussionist Don Moye. Trumpeter Lester Bowie, reedist Roscoe Mitchell (both of whom are respected soloists), and multi-instrumentalist Joseph Jarman round out a remarkably talented cast. Not all of "Nice Guys" is immediately accessible, but all of it is challenging and dynamic.

ALTO saxophonist Arthur Blythe has played with Lester Bowie and other AACM members, but his new album, "Lenox Avenue Breakdown," is markedly different from "Nice Guys."

Blythe gained prominence in Los Angeles, but was not part of the studio session scene. In fact, he's taken much of his style from his days with Gil Evans and Chico Hamilton. This makes Blythe's music more comfortable than some of the other "new jazz."

He uses basic swing and bop, as well as Latin rhythms, for a textured layer of sound. Blythe's supporting musicians include tuba player Bob Stewart, and the use of the jazz tuba adds a new perspective to the music. Its full but brassy bottom blends with the stellar rhythm section of bassist Cecil McBee and drummer Jack DeJohnette and helps push the pieces through their sometimes complex tempos.

Now that Blythe has a major label (Columbia) behind him, there's no telling how big an audience he can build for his unique brand of avant garde playing.

AND SPEAKING of audience, there was no better example of the new appeal of this kind of music than the sight of Milton Berle introducing Ornette Coleman on a recent "Saturday Night Live." There was Uncle Miltie, "Mister Television," announcing a man who might well be called "Mister Free Jazz."

Ornette Coleman is close to 50 now, but his music is youthful and constanly growing. Coleman's latest album, "Body Meta," was recorded for John Snyder's Artists House label. Snyder used to work for A&M's Horizon Records, which specialized in quality jazz. At Artists House, Snyder has given the musicians free rein, and the result - at least in Coleman's case-is a boundary-expanding effort.

On "Body Meta," Coleman uses two electric guitars (Bern Nix and Charlie Ellerbee) to play off each other and the rest of the ensemble. Some of the music sounds like intergalactic disco; other passages are a bit frenzied, with Coleman's alto saxophone riding roughshod over the furious rhythms. There are also lucid passages of warmth and relaxation, though there is never a predictable moment. On the whole, "Body Meta" may be Coleman's best work in years and shows that even the veterans of the jazz fringe are still in the thick of things.

ANOTHER veteran of the loft, Sam Rivers, is also coming on strong. Rivers has a new quartet featuring bassist Dave Holland that will release its album "Waves" any day now. The quartet, which appeared at D.C. Space last month, is equally capable of gentle but demanding ensemble work and quiet trade-offs between Rivers' reeds and Holland's bass. Rivers and Holland have also teamed up for two volumes of duets by pianist Paul Bley's Improvising Artists label, and these works, though they can become tedious, also offer rich examples of just how hypnotic the more experimental playing styles can be.

All of these artists have managed to break out of their cult modes without sacrificing their artistic principles, and now major record companies and bigger audiences are picking up on music that won't ever sell platinum but will provide artistic alternatives.

And how do the musicians, some of whom have reputations as being unconcerned with the masses, feel about the apparent surge? Arthur Blythe probably sums it up best:

"The lofts are great; they served as a showcase for me and played a part in my development, and I like their intimacy. But I'm ready to start playing larger halls. I want as many people as possible to hear my music." CAPTION: Picture, THE ART ENSEMBLE OF CHICAGO: STILL CHALLENGING, BUT LESS OBSCURE.