Many gambling stocks took a tumble yesterday as the New York and American stock exchanges maintained or tightened margin requirements on the high-flying issues, and a major analyst issued a cautionary research report about the prospects for legalized gaming in Florida.

Resorts International, which has the only operating casino in Atlantic City, saw its stock ride a treacherous roller-coaster. The class "A" shares closed down 10 points to 180, after trading as high as 208. A week ago the stock closed at 123 1/2.

After the close of the market, the American Stock Exchange announced that the margin requirement on the Resorts "A" shares would be raised from 75 percent to 100 percent, meaning that an investor must put up all cash to buy the shares.

The thinly traded class "B" shares already are subject to 100 percent margin. They closed at 265, down 1 for the day, after trading as high as 325. Resorts "B" finished last at 150.

The NYSE after the close of trading announced that its 75 percent special initial margin requirements would be extended on six casino-related stocks until Sept. 29. The affected stocks are Bally Manufacturing, Caesar's World, Harrah's, Playboy Enterprises, Ramada Inns and Del E. Webb. The margins had been raised from 50 percent to 75 percent two weeks ago as the feverish investment in these issues continued.

The possibility that Florida voters may turn thumbs down to legalized casino gambling in a Nov. 7 referendum, was meanwhile raised in a special wire to Merrill Lynch, Pierce, Fenner & Smith customers from Harold Vogel, the firm's entertainment stock analyst who has been following the gaming issues.

Much of the action in a wide range of hotel and gaming issues recently has been based on the presumption that legalized casino gambling will inevitably spread to a number of states besides Nevada and New Jersey as state governments look for new revenue sources in the wake of a wide-spread taxpayer revolt. And the next referendum is scheduled for Florida.

"Speculators in the casino stocks appear in many cases to have assumed that Florida legalization is a foregone conclusion, " Vogel said, but noted that a recent straw poll by the Miami Herald showed that if the election were held now, it would be rejected by a margin of 5-1-to-43 percent, with 6 percent still undecided.

Another poll by the Orlando Stentinel Star showed that 52 percent of voters opposed the referendum while only 39 percent supported it.

Vogel said "the Florida vote may well be regarded, in our opinion, as the important test of how quickly casino gaming may spread to other states." And he added that, as election day nears, stocks could gyrate with the polls.

The Merrill Lynch analyst reinterated his view that over the next one to three years "earnings growth potential is likely to be above average" for "the largest and most experienced operators" of casinos. And he recommended "long-term investment in the leading companies" while warning that "speculative purchase of the many relatively unknown and unseasoned new entrants in this group is fraught with risk," But "regardless of pedigree, all casino-related stocks may - as the Nov. 7 Florida vote on legalization of gaming in Miami Beach nears - be approaching an important crossroads with respect to their medium relative performance potential."

For a number of stocks that crossroads may have been reached yesterday, after the report circulated, though not all issues suffered declines.

Ramada Inn, the volume leaded on the Big Board with a turnover of 2.2 million shares, closed up 1 to 16. And Caesar's World gained 2 to 62 1/2.

But Holiday Inn dropped 3 1/4 to 29 1/4, Howard Johnson was down 5/8 to 15 1/8, Del E. Webb lost 1/4 to 35, Bally Manufacturing dropped 2 3/8 to 62 7/8 and Golden Nugget was down 3 to 30 3/4.