"You must be going to Versace," said the building porter as he wiped the white marble walls of the Madison Avenue office building. "Today, everyone coming here goes to Versace."

Indeed, only three weeks after he had shown his fall designs in Milan, buyers from all over the country were tracing Gianni Versace to his New York showroom. It had been open all of one week.

Versace was the designer who led the military fashion march down the runways at Milan's designer showings. In Milan, too, Versace booked sizeable business with American stores for his sportswear, leathers, knits and dresses. "He is the Italian designer who understands the American customer best," says Ellin Saltzman, fashion director for Saks Fifth Avenue.

In six years, Gianni Versace has built up an impressive volume of $20 million worldwide. "I would like to be in America what Geoffrey Beene is in Europe," says Versace, referring to the enormous success in Europe of Beene, the first American to do so.

So after Milan, Versace combined his four collection - Complice (sportswear), Callaghan knit), Genny (evening wear and blouses) and Versace (leathers and couture) - into one fast-paced, jam-packed presentation at the Pierre Hotel. Buyers who missed the Milan shows flocked to see what the hoopla was all about. And those buyers who had seen the shows in Milan came to confirm their first impressions.

Few seemed disappointed.

Versace, 30, tucks himself comfortably into the windowsill of his freshly painted white office. He's wearing a wrinkled lined jacket with sleeves pushed up, a ceased linen shirt of his own design. His tie is loosened. If he weren't in the fashion business, you might accuse him of looking "unkempt."

His accent is heavy but that doesn't slow down his English. It's almost too smooth, in fact, for someone who insists he just "picked up words" on visits here. His answers are pat; he's been asked some questions so often he has refined his answers.

"All people want to have uniforms," he begins. "It is easy, it saves time to wear uniforms. You get up. You take no time to select, to choose. It is quick," he says. "Look, I wear a uniform, always the same, and your vest, shirt and skirt," he adds, pointing to his visitor, "that is a uniform, too."

Versace adds intently, "Uniforms are good for people. Like jeans."

His military uniform style with broad shoulder jackets and coats and military decorations in his sportswear collection are "from the mood that is like war now in Italy." But people are tired of the lawlessness, he says. "On the next elections, people will go in for a different line."

If the clothes appear aggressive at first glance, suggests Versace, take a second look. "You will see the fantasy, the soft blouses, the flowers, the scarves, the jewelry."

He carries the military style in greater variation than most designers. There's the African look with military jackets over skirts of leopard chiffon; a Scottish military look is done with tartans, plus all the military accessories in belts, berets and medals.

He calls his white suits for evening "grand uniforms of the night and does them with tuxedo or spencer jackets, sometimes decorated with epaulets, lariats, ribbons and medals.

Versace says he was "born in fashion." His mother, still a severe critic, had a boutique and custom design atelier caller "Elle" at Reggio in Calabria and he started early as a buyer for her. He studied architecture in southern Italy but was lured into the knitwear business on a visit to Milan.

It is very close, architecture and fashion," says Versace. "Good fashion is pure form. Not complicated," he says. "Americans (like Stephen Burrows and Geoffrey Beene) make good fashion because it is real. French designers make too much fantasy. They are not clothes for customers.They are for actresses.

Versace claims to do the biggest ready-to-wear business in Europe of any Italian designer, but now he is anxious to develop more trade in America. "It is not enough to be good and important in Europe," says Mirella Petteni, a former model who now runs the Versace enterprises in America. "To be successful in America - that is the real exam. If you make it here, you really have made it.%

Says Anita Gallo, vice president of B. Altman and the first to have the Versace designs in America. "His things are contemporary classics; the woman who bought this things four years ago is still wearing them and can just add pieces to update them as she wants." Wendell Ward, vice president of Garfinckel's, where the Versace designs were featured in a recent benefit fashion show for CARE, goes further. "He's the Italian Yves Saint Laurent. He's full of ideas and many of his ideas, particularly his tucking, pleating and shirring that gives fullness to clothes have influenced other designs that we have in the store. "Woodward & Lothrop and Saks-Jandel also will carry Vesace designs in Washington and Saks Fifth Avenue will open Versace boutiques in seven stores.

Versace already makes menswear, and has contracted a license to design blouses for the American manufacturer Ciao.

Perfume next? "Yes," says Versace, putting up his palms as if to stave off a push. "I must go slowly. One step at a time."