The man who has sold America more art books than anyone else sits at his round, glass worktable and laughs about competing with his own name.

Harry N. Abrams surprised Bennett Cerf and other publishers in the 1950s by not going broke estimating that Americans would support a company publishing only art books and it out to prove he can do it twice.

His first success, with a company named after him right down to the middle initial, didn't come easily. Abrams remembers that Cerf, who headed Random House, bet him $100 that he would be publishing other kinds of books with his fledgling company within three years.

"I'd have been glad to pay the $100 and go on to something else if it would have helped," Abrams says of the first five years of his companY. "Every day we were fighting not to go bankrupt."

The second time around, Abrams is delighted to report, is easier.

He sold Harry N. Abrams Inc. to the Times Mirror Company in 1966, but remained as its head. However, in 1976, with Abrams entering his 70s, the conglomerate thought it was time for the founder to be only chairman and let younger men run the business.

"They did what they had to do and I did what I had to do," Abrams says. It was a friendly parting and Abrams is happily preparing the first nine books to be published by his new art book firm, Abbeville Press. "We're doing more business today than my old company was when we sold it to Times Mirror," Abrams says proudly. Several of his first books are sold out before they have even been printed.

"The booksellers are wonderful. They've bought the books as soon as they learned that it was Harry Abrams who was doing them Sight unseen," Abram says.

Abrams is forbidden to mention his own name in advertisements for 10 more years under the terms of his arrangement with Harry N. Abrams Inc., so the booksellers learned by word of mouth. But they learned quickly.

Abrams is proud of the quality of reproductionin his new list as well as the books he did with his first firm. "No one is producing this kind of thing," he says, pointing to proofs of a forthcoming book about Pompeii.

Not even his old firm? Abrams laughs and declines to answer. "I understand they are doing very well now," he says.

The man whose books dominate art book sections in stores around the country is short, direct, friendly and 73 years old.

"I wish I'd made this appointment for earlier," he says, explaining that he gets to his office early in the morning and is tired by the late afternoon. Abrams rubbed his eyes as if to demonstate fatigue and that was the last sign of tiredness for more than an hour.

The problem with producing art books, he says, is to bring the per-copy cost down by selling large numbers. If you're good at the formula it works as easily as one of Catch-22's Milo Minderbinder's. All you have to do is sell enough books so that you can sell enough.

Enough, of course, is a hard number to define. Like a good politician who is careful not to hire too large a hall lest the empty seats make him look bad, Abrams would rather sell out all 25,000 copies of a book than print 50,000 and have 15,000 returned by the bookstores.

Bookstores don't remember the book they sell 600 times if they have 200 left over, Abrams said. The one that was a best seller was the one they had 50 copies of and sold out. "It's all psychological."

With dismay, Abrams tells the story of a book by photographer David Douglas Duncan that was selling like hotcakes at $35 a copy until Duncan urged and got a large second printing just about the time the book was exhausting its market. "Duncan could have been a hero," Abrams says. "No one remembered that the book sold thousands, they just saw the remaining copies marked down to $5.95 in all the stores."

Another way to make money publishing art books - or any books - is to hold costs down.

One author who has dealt with Abrams describes him as "tempered steel in a velvet glove." He'll tell you that he's going to lose money on the book anyway so he can't write a royalty into the contract, this man says. "That's a typical Abrams contract - no royalty."

But even Abrams' anonymous critics bruised from dealing with him praise him for publishing books on modern artists who would otherwise have been ignored. "He made a terrific difference for artists," one said.

1950, the year that Abrams left his job at Book of the Month Club to start his first publishing business, by coincidence was also the year that the Swiss art book Skira decided to enter the American market.

Abrams published three books - on Renoir, van Gogh and El Greco. Skira sent three books to America and Abrams remembers that the booksellers thought six art books was a lot of art books to handle.

He also remembers that as he became known as an art book publisher many people would say to him: "Oh, you're the fellow who publishes those Skira books."

Today there is no Skira and Abrams is starting his second company. "It's a tough business," he says. "You have to watch everything."

One of the things Abrams has always watched carefully is the quality of his reproductions. He has been criticized for reproductions that lacked color. "Hell, the original isn't colorful. The artist did a somber painting," Abrams says.

Once he went to Europe with a colleague and was looking at a Rembrandt that was heavily red.

"If we did a faithful reproduction of that," his colleague remarked, "everyone would say "What a terrible plate."

Although Abrams collected on his bet with Cerf, about 15 years after he began his company he was doing very well with a number of titles that had nothing to do with traditional art books.

Abrams' art collection - which fills his large New York apartment, his country house, his two sons' houses, his office and a considerable storage area - includes 20th-century European masters like Modigliani, Chagall and Picasso and is one of the most important private collections of pop art.

Elvis Presley and a flower painting by Andy Warhol, a large Botero, a giant Morris Louis and a football player by Wayne Thiebaud hang in his office. Abrams' collecting taste has influenced his publishing decisions, but while most of the books he has done on living artists have lost money, a book on another of his collecting passions - seashells - did handsomely.

"They made Michelangelo, Donatello, Henry Moore and all the great sculptors look like little ants," Abrams says of the beauty of shells. "They're God's sculpture."

Abrams also had huge success with books about two artists - Walt Disney and Norman Rockwell - who were not much thought of in the world of museums and galleries.

"Some art critics are going to pan me for publishing this," Abrams told an interviewer in 1970 as his Norman Rockwell book was about to appear. "They'll say he's a commercial artist. Maybe he is - but he's got something much more - a spark of magic."

Magic art or not, the book had a magic appeal. At $75 it sold 65,000 copies in its first year.

Abrams pointed out that a number of Norman Rockwell books had been published before his without such phenomenal success. "Good ideas can fall flat," he says, "and mediocre ideas can become a flash - it's all how you carry them out."

Abrams has no easy secret for how you sell art book sin America. "You don't do it yourself. You do it with the help of the man upstairs," Abrams said pointing through his 15th-floor Park Avenue ceiling. "I've become very religious about these things."

When asked a second time what someone would need to become a successful art book publisher, Abrams says: "I just heard that Nelson Rockefeller is going to publish some art books. With his kind of money it wouldn't be hard."