Julian Schnabel, who within four years became New York's most talked-about young artist, packed up his broken-plate paintings this week and moved them from the downtown Mary Boone Gallery to the uptown Pace Gallery. The art world was abuzz.

Mary Boone, the jilted dealer who deserves much of the credit for turning Schnabel into a household name, tried to remain philosophical in the wake of her loss.

"I always loved his work and enjoyed having it here and exhibiting it," Boone said from behind a Louis XVI gilt desk in her newly expanded SoHo headquarters, "but Julian is not the kind of person you talk into doing something he doesn't want to do. He didn't want to show in a gallery with other artists. It would be distressing for an artist to go from most important to being part of a group."

Leo Castelli, the dean of contemporary art dealers, interviewed in his SoHo gallery, was visibly upset and angry. Castelli, who also showed Schnabel's work, lost himto his chief rival, Pace Gallery president Arnold Glimcher. It was not his first defeat at Glimcher's hands.

In 1980, the Whitney Museum of American Art paid $1 million to acquire "Three Flags," Jasper Johns' lush pop-art painting of the Stars and Stripes. Glimcher profited from the sale. Castelli, who represents Johns and originally sold the painting in the 1960s for $900, did not make a penny from the sale.

"I have nothing to say about this," Castelli said. "Am I upset? You'll have to ask Mr. Schnabel if I am upset."

Upon learning Schnabel left the Boone Gallery, Castelli called him on the phone: "I have nothing but contempt for you," he told him, according to sources, and slammed down the receiver.

Art maven Barbara Jacobson, who has collected Schnabel's work from the beginning, and counts Castelli among her closest friends, shot off a telegram to Schnabel denouncing him.

Glimcher, who represents the work of many top-flight artists, including the estates of the late Pablo Picasso and Mark Rothko, was, of course, delighted by his latest acquisition, as was Schnabel by his new dealer. Said the artist in explanation of the shift: "It's spring. That's why I made the change."

On the face of it, Schnabel's switch to Pace is a smart move. Although Glimcher has been accused by other dealers of having more business acumen than taste, he has an unmatched track record in promoting and marketing the works of established artists. He spares no expense to provide a glossy backdrop for their work in his 57th Street I.M. Pei-designed gallery. Since he won the Rothko estate from the Marlborough Gallery in a courtroom drama that ended with the latter convicted of mismanaging it, Glimcher has shrewdly pushed the price of Rothko's work to a half-million dollars. He refuses to discuss his finances, but most of his artists maintain a monthly income based on an annual guarantee of a set amount.

On the other hand, Schnabel and Boone have been inexorably linked since the beginning of both of their respective careers, sort of the art world's Woody Allen and Diane Keaton. After four years of relentless publicity, Schnabel's prices have soared from $3,000 for one of his broken-plate paintings to at least $93,500, the price paid last May at Sotheby's for "Notre Dame" (1979). Though Boone represents other artists--David Salle, Gary Stephan, and, most recently, Eric Fischl and a number of young German painters brought in recently by her friend and partner Michael Werner--she is best known for her handling of Schnabel's career, and for her style.

Why did Schnabel leave?

The art world, which thrives on gossip, has had a heyday with this one. Some in the art community say that Schnabel was in need of money to finance property investments on Long Island and in Manhattan, and that Glimcher offered him a million-dollar annual guarantee of sales. Others say he was resentful of Boone's new artists. She opened her expanded gallery this month with an exhibition of the work of German neo-expressionist Georg Baselitz, rather than Schnabel.

"This is not Hollywood," Glimcher responded when asked about the million-dollar guarantee, and refused to divulge the details of his deal with Schnabel, as did Schnabel.

"An ernormous hype surrounds his work," Glimcher added. "He would like to withdraw from that scene . . . away from the madness of that scene."

"My gallery has nothing to do with hype," Boone responded. "Hype is an invention of the media . . . I have never and do not now court the press."

"I think Pace is a more stable situation," Schnabel responded, ". . . whether it is Mary plotting or me plotting . . . now we'll know. The Germans artists might have expedited my leaving. I think there are other situations that also expedited it."

When asked whether he had threatened to leave last fall if Boone represented young Eric Fischl, Schnabel hesitated and said: "I wasn't supportive of his work. But, in the end, I told Mary I wouldn't stand in his way."

When asked whether he had considered exhibiting his work at Leo Castelli's gallery, rather than at Mary Boone's, Schnabel said: "Leo is too busy for me. He has lots of other commitments."

Where did Schnabel and Glimcher first meet?

"On a tennis court on Long Island last summer," Schnabel recalled.

Said a 57th Street dealer who asked to remain anonymous: "The amazing thing is that everyone is talking about Schnabel again. And he hasn't done anything good, if he ever did, in over a year."