At the age of 36, Robert Brennan has already amassed two fortunes. Now he is starting to work on his third.

He founded First Jersey Securities with a three-person staff seven years ago, and its spectacular growth established him as one of the most successful and controversial men on Wall Street.

Then Brennan started a restaurant corporation that operates 16 establishments from New York to Florida.

Now Brennan has found a new investment opportunity that makes him exclaim, "I see a gold mine! The potential to make profits is gigantic."

The business that starts his capitalistic juices flowing is horse racing, and he has made a spectacular entrance into it. During the last 10 months, he has bought more than $10 million worth of horseflesh. Last March, he didn't own a single thoroughbred. Today, he has 58 horses of racing age, 20 yearlings, 21 mares and shares in 15 stallions.

Skeptics are already questioning some of his moves and remembering other wealthy men who entered racing in a splashy fashion and wound up losing millions. Brennan has faced a similar legion of doubters and detractors in the securities business.

Some critics call First Jersey a "boiler room" operation. They say Brennan buys cheap stocks with virtually no assets, hires high-pressure salesmen to peddle them to unsophisticated customers and manipulates the market in these stocks. He was scrutinized intensely by the Securities and Exchange Commission and by the National Association of Securities Dealers, but emerged with his reputation and that of the company largely unscratched. First Jersey has 100,000 customers, and Brennan feels ready to conquer a new world.

He feels very much the same entering racing as he did when he went into the securities business. "I wan one of nine kids growing up in Newark in a five-room apartment," Brennan said. "On Wall Street, my peers and competitors were men who had been handed down a few million by Uncle Joe after they graduated from Harvard or Yale. You have the same situation in the Sport of Kings': it's all at least second-and third-generation money."

Racing's old guard may raise its eye brows at Brennan's spending binge, since the prevailing wisdom says that this is a sick industry. Race track attendance has been declining in recent years; few horses can earn enough even to pay for their upkeep; most owners get into the business to establish a pleasurable tax writeoff.

Brennan argued, "Every industry goes through peaks and valleys, and racing is in one of its valleys. But that's the time to enter any investment: you buy when things are out of favor."

Brennan sees two trends that will create a boom in racing. One is the increased involvement of businessmen (as opposed to old-guard 'sportsmen') in racing. The other is the inevitability of public ownership of horses. He envisions the day when the public will be able to buy shares in thoroughbred operations, and thinks it will create tremendous new interest in racing.

"In the securities business," he pointed out, "it was once unheard of for a brokerage firm to be publicly owned. But when Merrill Lynch went public, investors loved the opportunity to own the firm they do business with."

Brennan expects to be a trailblazer in the public ownership of thoroughbreds. In the meantime, he is building his own Due Process Stable (a name he says was inspired by the lack of due process he received from the "crazy, dumb bureaucrats" in Washington). His purchase have ranged from $20,000 for a colt he named Newkidontheblock to $1 million for the mare Alma North. His critics may call him a lot of names, but nobody will ever call him a piker.

"I know a lot of guys come into this business, shoot their load in the first year and go down," Brennan said. "But you have to have staying power. You have to make sure you have the financing and capital so you can go through tough, dry periods."

Brennan is prepared to endure these droughts, but he also fully expects to make a large amount of money in racing. If his performances are any guide, he will.