In Cherry Hill, N.J., this week, architects were huddling over their blueprints and pile drivers were starting to work on what is now a desolate tract of land. This was the beginning of a construction project that people in the racing industry will be watching with a mixture of suspicion, fear, and amazement.

Garden State Race Track is being rebuilt on the site of the old facility that was destroyed by a fire in 1977. In an era when a number of racetracks have gone out of business, and most others have trouble raising money for modest capital improvements, this $100 million project is going to create one of the most spectacular tracks in the world.

The man responsible for this formidable undertaking is Robert Brennan, an enthusiastic, clean-cut young man whose face has become familiar in the television ads for his company, First Jersey Securities.

But Brennan has detractors in the investment community, many of whom have questioned his business tactics. In recent years he has had trouble with both state and federal securities regulators. Several months ago, in fact, the Securities and Exchange Commission filed a civil complaint against First Jersey Securities over its trading in a small oil and gas exploration firm.

Brennan has denied that his firm has violated securities laws.

Because of all the controversy that has surrounded his company, Brennan was viewed warily by racing people when he started buying horses a few years ago. This was, for him, the fulfillment of a boyhood dream. He remembers betting his paper-route money with a bookie when he was a kid growing up in Newark, and always loved to go to the track, but never had a chance to do it much while he was busy amassing his fortune.

At first Brennan viewed racing as a hobby. But quickly he saw that it could be a profitable business, too. He started a company, International Thoroughbred Breeders, that entered the commercial breeding business. And then he started thinking about the possibilities of the defunct Garden State track.

"I looked at Garden State in 1980," Brennan said, "but economics made buying it impractical. No rational person could make the capital commitment. But in 1982 the state legislature passed legislation which provided tax incentives and increased the number of racing days to 200. It was still venturesome, but I believed the track could be an economic success."

Brennan offered to buy the property for $15.5 million from the trust representing the estate of the late Gene Mori, who had owned Garden State. Other prospective buyers fought him vehemently. "Mud was hurled from every direction," he said. But he got the property, and then he set out to raise the money he needed. And nobody can raise money like Bob Brennan.

He issued a prospectus for a secondary offering of stock in International Thoroughbred Breeders, and his other arm, First Jersey Securities, set out to sell it. The stock offering was effective at 9:55 a.m., and at 10 a.m. it was sold out. Brennan had raised $93.5 million in five minutes from investors he characterizes as "young professionals, business people, clients who are more or less risk-takers." With their money he had the ability to build a racetrack like no other.

The grandstand will be built around a glass-enclosed atrium paddock, with balconies overlooking it from all seven tiers of the track. Cafes and restaurants will be scattered throughout the plant.

Garden State is scheduled to open Feb. 1, 1985, and conduct a five-month thoroughbred meeting; from September through December it will offer harness racing. Naturally, its prospective competitors are apprehensive. Officials at Keystone Race Track have to be quaking, because Garden State will compete with them head-on for fans in the Philadelphia area. Maryland racing officials are plainly worried, too, because the New Jersey racing circuit will be very attractive to top racing stables.

"I don't believe our success will have to come at the expense of others," Brennan said, "but in any event we couldn't approach this project by trying to achieve mediocrity. We made a decision to pursue excellence in every sense of the word. We're going to put up an extraordinary facility that will be exciting and glamorous and attractive to people who've never been to a racetrack. And if we're successful, we may stimulate other people to do the same."

Although he sees his project as a potential stimulus to the whole American racing industry, he is not surprised that people in the sport resent him or hate him.

"One trouble with racing is the people don't feel joy for the success of others," he said. "That's manifest at the races; people are jealous when someone else wins a race. It's not unlike the atmosphere on Wall Street 15 years ago, when the brokerage firms were run by old families. They needed new blood, new capital and new excitement, and that made Wall Street spring alive. The same thing is going to happen in our industry."

If his analogy is apt, and Brennan has the impact in racing that he has had on Wall Street, people still may not like him, but they are going to have to deal with him and his success for a very long time.