HE HAS BEEN called America's most inventive designer and classed in the league of Balenciaga, Madame Vionnet, Chanel and Cardin. A mad genius who created dresses and coats with the eye of an architect, he perfected cuts and construction that continue to influence the fashion business.

He dressed Gypsy Rose Lee and Jennifer Jones. Austine Hearst, Babe Paley, Dominique de Menil, Elsa Perretti and Millicent Rogers were among his best customers. His designs boosted many to the International Best Dressed List.

Christian Dior credited him with inspiring the "New Look." And when Paris couturier Paul Poiret saw his shapes, he is supposed to have said, "Charles James, I pass you my crown."

Charles who?

In spite of two Coty American Fashion Critics Awards (which he returned as "too commercial"), and awards from the wool industry and Neiman-Marcus, Charles James, who died in New York last week at age 71 of angina complicated by bronchial pneumonia, was even to many in the fashion business, a total unknown.

He was a designer's designer, and for a small group of disciples, patrons and friends he was a cult figure whose inventiveness and demand for perfection set him apart.

(New York cable television station is offering a 36-part documentary on him.)

But in later years, his personality became so disagreeable, even to those who were preparing exhibits honoring him, that his well-earned credit as creator of the wrap culotte, the spiral seam dress or the angular zipper, as well as his reputation as the forerunner of many Paris styles, never got much attention. That only angered him more.

One costume specialist simply avoided showing his clothes rather than risk his wrath, possibly even harm, if the presentation was not to his liking.

"He spent all of his energy and time trying to attack people who had done him wrong," says Dominique de Menil. "Unfortunately he was so aggravated that it took too much of his creativity."

Austine Hearst adds, "I can imagine the terrible rumpus he's creating in heaven right now because he died while the (New York) press was on strike."

James was born in Sandhurst, England. His father was a colonel, his mother a Chicago heiress. He told Polly Logan that when he was seven he used to go with his grandmother from Chicago to Paris and would sit in on the fittings she had with Paris couture designers. "His grandmother's liking for beautiful clothes and sense of fabric and color started his interest in fashion," Logan said.

He was educated at Harrow in preparation for Oxford; but when he refused to enroll, he was dispatched by his family to Chicago to work with utilities magnate Samuel Insull. Electronics didn't suit him much better than the family profession of the military, so he opened a millinery shop on Oak Street.

In 1929, encouraged by old school chum Cecil Beaton, James moved to New York, and established a business selling custom-made clothes. He would occasionally make different versions of the same dress for three or four different clients and, in the manner of the French couture, sold copies of these designs to department stores. Best and Co. was the first to buy his hats and ran a full page ad to boost their sales.

James opened a London salon in 1931 where some of the Bloomsbury set, including Lady Duff-Gordon and Lady Diana Cooper, became his clients. Eleanor Lambert remembers going to see him in London: After a long wait she was admitted to his back palor where he was stretched out, Recamier style, on a chaize lounge totally engulfed in tuille. He later moved to Paris where he was surrounded by wealthy clients and was a friend of Gertrude Stein, Jean Cocteau and Virginia Woolf. Both Coco Chanel and Elsa Schiaparelli chose his clothes for their private wardrobes.

In the 1940s and '50s, American women who demanded dramatic clothes sought out James in his 57th Street salon in New York. Polly Logan remembers the pint satin figure forms he kept in the back room, and how he constructed clothes on the body like a sculptor using fabric, starting with a modern corselet of whalebone that pushed the client into shape.

"His dresses healed every flaw," says Austine Hearst, recalling a Dubonnet colored satin Victorian gown with a huge white organdy pleated underskirt - which forced the men sitting next to her at dinner to sit sideways on their chairs.

Such dresses, built rigidly with the seams as girders, could literally stand on their own and were a challenge to travel with.When "Bootsie" Hearst went off to the coronation in England, she took her dresses in a black coffin-like box. Even getting to a ball was a trial. Women rented rooms at the Waldorf Astoria and had their dressed delivered there so that they would have to go no further than the elevator. Travel by limousine or taxi simply was too difficult.

A consummate perfectionist, James is said to have spent three years and $25,000 to develop the perfect armhole. He used calipers instead of a tape measure; and if he felt a dress wasn't exactly right, he'd rip it right off the body of the client being fitted and she might wait for months, even years for a new one. ("He did that three times with one dress," recalls Florence van der Kemp. "Of course, I paid for the fabric.")

Occasionally he'd decide that the dress he made for one woman would be better on someone else. He'd ask for the dress back on the pretense that he needed it for an exhibition, then sell it to the other party. "Imagine some lady rich enough to afford his clothes willing to pay for someone's else's old dresses," says Austine Hearst.

He was equally perfectionist about his furniture. John de Menil, then chairman of the board of Schlumberger asked James to furnish his Philip Johnson designed house in Houston. One piece, a sofa shaped like lips was totally three dimensional, everything curved. "It drove upholsterers wild," recalls Domenique de Menil, a patron of his clothes, who remembers it took five successive artisans, each working six months to upholster the sofa to James' satisfaction.

Price tags for his custom made "grand entrance" gowns were in $1,000 to $3,000 range. "He managed to figure out just how much you could pay," says Augustine Hearst, who paid on the installment plan.

(James had a falling out with Gypsy Rose Lee over the bill he sent for her dress she wore in her very final performance. In it, she peeled off layers of a dress that fully clothed the five nude minions standing by. James figured the price based on five dresses, Lee considered it one. Lee supposedly won.)

In spite of the high prices, his career gradually began to decline and James spent what money he had fighting for recognition. He threatened lawsuits and sent 10-page letters of outrage, with carbon copies, to editors, clients and friends about credit he felt others were getting for his work. After a long and costly legal fight, he won one lawsuit against manufacturer Samuel Winston, whom he charged with pilfering his designs and selling them under another label. (Gypsy Rose Lee testified in his behalf.)

Twenty years ago Charles James closed his 57th Street salon and "retrenched," his words, at the Chelsea Hotel in New York, spending his full time working, teaching and drawing. He continued to make clothes for a few clients, Elsa Perretti among them, and created a vast number of erotic drawings which were exhibited at the Everson Museum of Art in 1975. And he kept writing the letters.

Halston, who owns a number of the erotic drawings, plus a share of letters, asked James to join him when he opened his Madison Avenue workrooms. "I felt sorry for him," says Halston. "He couldn't pay his rent." Like most such liaisons, the results were disastrous. And the business arrangement broke up angrily. Eleanor Lambert says that "He simply failed to understand that fashion is a problem of teamwork and that no article of fashion springs full-fledged from the brain of the designer."

If the fashion establishment turned its back on him, the art world didn't. Matisse ' and Tchelichew painted his shapes. Salvador Dali called the white satin quilted jacket he designed in 1938 the first piece of soft sculpture. The jacket was later bought for the Victoria & Albert Museum's costume collection for $6,000 - reportedly the highest price ever paid by a museum for a piece of clothing. Other James designs are at the Smithsonian and the Metropolitan and Brooklyn museums.

He was the only fashion designer to be honored with a Guggenheim fellowship and was chosen for his drawings, which Robert Motherwell said "are more powerful and more to the point than regular artists."