Attempts to put on a ballet the question of whether to hold the 1984 Summer Olympics here could force Los Angeles to withdraw its bid for the Games, warns a top city official.

Anton Calleia, special Olympics liaison for Los Angeles Mayor Tom Bradley, said a charter amendment proposed for the June, 1978, ballot by City Councilman Bob Ronka would make it impossible for the city to submit its bid a meeting scheduled for May 19, 1978, in Athens at which the International Olympic Committee (IOC) is to give its final approval on a site for the 1884 Summer Games.

"It's a real threat by all means. I think a referendum is the end of the Games in Los Angeles. It creates an undesirable atmosphere," Calleia said. "We could not in good faith go into Athens and put in a bid with a referendum hanging over our heads. We'd be the laughingstock of the world."

Ronka, who proposed the referendum on the Games along with a charter amendment barring the expenditure of taxpayers' money on the Olympics, dismisses Calleia's claim as tactics designed to stop his measures. "I think Calleia thinks the Olympics are his baby. He has an emotional attachment to it and doesn't want anyone to shift the blinders on him," Ronka said.

Moves to retstrict the city in its dealings with the IOC, the Switzerland-based group that runs the quadrennial Games, have increased since Sept. 25 when the United States Olympic Committee designated Los Angeles as its preferred site over New York City. Concern over the Games has grown even more since Nov. 1 when Lord Killanin, head of the IOC, announced that Los Angeles was the only city in the world now bidding for the 1984 event.

Ronka, and others seeking public votes on the Olympics point, point to the fiscal problems resulting from the 1976 Montreal Games as justification for their ballot initiatives. Olympic critics such as City Councilman Ernani Bernadi, who has proposed his own strict spending charter amendment for the elections ballot, claims the Games saddled Montreal with an enormous debt it has yet to pay.

"I'm not against the Olympics. I think it is a wonderful event, but I'm worried about it after looking at the financial fiasco which took place in Montreal." Bernardi said. "The only way you can guarantee this city's tax revenues is with an amendment to the city charter. No other way can we guarantee the city taxes won't go to this."

City and Olympic officials here have pledged from the beginning that the Games would, once the final recipts are counted, not cost the pity taxpayers a dime. The budget submitted by the Los Angeles Games' sponsor, the privately financed Southern California Committee for the Olympic Games, calls for $185 million in expenses balanced by revenues from tickets, television rights, contributions and other sources, including federal funds. A net Olympic surplus of $750,000 is projected by the Committee.

John C. Argus, president of the committee, says comparisons with Montreal are inaccurate because Los Angeles, unlike the Quebec city, does not have to build either an Olympic stadium or an Olympic village, both of which are extremely costly. Argue proposes to use the 92,000-seat Los Angeles Colisium for the main opening ceremonies, track and field events. Dormitories at UCLA and USC, he says, will serve as the Olympic villages.

"Montreal is casting a shadow over us. People are always asking us whether we are going to have another Montreal. I'm sick of it to tell you the truth," Argue said. "It won't happen here - the big difference is we have the facilities already, they didn't."

Despite city and Olympic officials' claims that this will be "a Spartan Olympics", Ronka, Bernardi and others maintain that whatever the estimates are now, they may change by 1984. They say a city charter amendment forbidding the use of tax monies is the only way to make sure the city keeps its pledge.

City officials hope that the Ronka and Bernardi measures will fail to pass the charter amendments, Ronka and Bernardi have ps "an uphill fight" with only five of the Council's 15 members now likely to it.

But even should the Council fail to pass the charter amendments, Ronka and Bernadi have pledged to fight for passage by initiating a petition drive to put them on the ballot. Already a new group, Concerned Citizens-Olympics, has emerged from 23 neighborhood and taxpayer groups in preparation for a possible petition drive on the charter amendments.

One of the organizers of Conceerned Citizens-Olympics is David Mixner, a key figure in the Citizens for Colorado's Future, the group that ran the successful initiative drive in November, 1972, blocking the 1976 Olympic Games in Denver. Ronka consulted Colorado Gov. Richard Lamm, who headed the Citizens for Colorado's Future, before proposing his Olympic charter amendment last month.

All these events apparently have caught the New York-based U. S. Olympic Committee unaware.

"We can't say anything, there's nothing we can do. There's no other city to consider," conceded U.S. Olympic Committee spokesman Bob Paul. "We've never been faced with a possibility like this for the Summer Games before. This wouldn't happen in some other country where they don't have our pure democracy."