During summers in the Depression, young Clifford Perlman started his first business, setting up beach chairs by the Boardwalk in Atlantic City.

Now Perlman, 54, has another business on the Boardwalk. And the State of New Jersey is thinking about taking it away from him.

The New Jersey Casino Control Commission says it will vote here Thursday on whether to give Perlman and the company of which he is chairman, Caesars World Inc., permanent licenses to run the Boardwalk Regency casino-hotel in Atlantic City.

Its vote will follow five weeks of hearings in which the state's Division of Gaming Enforcement, New Jersey's casino investigators, presented details about Caesars' and Perlman's business transactions with reputed organized-crime figures.

If Caesars is not licensed, then a "conservator" named by the state would take over the Boardwalk regency and sell it, with the profits going to the company.

The decision is crucial not only to Caesars but also to New Jersey law enforcement officials, who have gone all-out to try to ban Caesars from Atlantic City. In addition, casino stock analysts say banks and lending institutions considering Atlantic City investment are extremely interested in the outcome, fearful that the state will be too tough on casino license applicants with "borderline" connections to underworld figures.

First, Caesars has a considerable investment at stake.

In the fiscal year that ended July 31, the Los-Angeles-based company took in $239 million in gross revenues and $16 million in profits from its 86 percent interest in the resort town's second casino-hotel, which was opened in June 1979 on a temporary license.

The company, listed on the New York and Pacific stock exchanges, reported total revenues of approximately $520 million in fiscal 1980, which also included revenues from its Caesars Palace casino-hotel in Las Vegas, Caesars Tahoe casino-hotel in Stateline, Nev., resort hotels in the Poconos and a small computer company.

Caesars also has said it wants to expand its 504-room hotel in Atlantic City and to build a second casino-hotel complex there with as many as 1,500 homes.

Some New Jersey law enforcement officials have their own reputations riding on the decision, having promised they would get allegedly shady casinos barred from Atlantic City.

Gov. Brendan T. Byrne and Attorney General John Degnan toughened their resolve to keep New Jersey casinos free of underworld taint following the resignation in February of Casino Control Commissioner Kenneth MacDonald, who admitted being present during a conversation in which a state senator was offered a $100,000 bribe from FBI agents in the Abscam probe.

Byrne then overhauled the five-member commission, naming four new members. To win a license, Caesars must get an affirmative vote from four commissioners.

The division of Gaming Enforcement recommended against giving licenses to Caesars World, Perlman, Chief Operating Officer William McElnea and board member Jay Leshaw because the division said they lack "forthrightness" and show a propensity for dealing with "unsavory characters."

The division focused on a range of "unsavory" relationships, including Caesars' and Perlman's real estate investments in Florida and Pennsylvania during the 1970s with Miami lawyer Alvin L. Malnik and Miami Beach hotel magnate Samuel Cohen, both reputed associates of underworld financial wizard Meyer Lansky.

Perlman said he regrets these deals, calling them a "huge mistake," and said he wouldn't invest with these men again.

"I've learned I've got to lean over backwards in any association I have," Perlman testified. "It's been a frightening experience for me."