Under a tent festooned with red balloons, Mayor Bill Neild helped break ground for a new downtown art museum in this city of 120,000 last week, declaring, "We're looking forward to milk and honey in a year and a half or so, when this is completed."

The mayor may have been indulging in hyperbole: Beaumont lost $20 million -- one-fifth of its city budget -- to E.S.M. Government Securities Inc. of Fort Lauderdale, Fla., when it failed March 4. E.S.M. did not have the securities it should have been holding as collateral on a loan from Beaumont.

A continuing investigation by the Securities and Exchange Commission indicates that a dozen or so municipalities and five savings and loan associations will lose about $315 million as a result of E.S.M.'s failure. The losses also forced the closing March 15 of 70 privately insured savings and loans in Ohio, and most of them remain closed.

Beaumont City Manager Karl Nollenberger, who has cut his own pay by $24,000, figures the city can survive fiscal 1985 by freezing hiring and salaries, borrowing from a state board to complete water projects, postponing a $32 million bond sale and major street repairs and, in a violation of the city charter, starting the next fiscal year $2.3 million in the red.

The deficit, its first since 1982, will have to be approved by the Texas Legislature, which is to adjourn May 27.

Meanwhile, a city employes' group has called for the resignation of Nollenberger, 37, who was brought here in 1983 to help bail the city out of a $7 million budget deficit. The finance director has quit, the city has initiated an internal investigation and a representative of the state attorney general's office is arriving Monday to look into possible violations of civil statutes.

"I think everyone in the country with the exception of the Mothers of Twins Club is doing an investigation" of the fallout from E.S.M.'s failure, said Tom Combs, a former Beaumont City Council member. "It just makes you want to pull your hair out."

Before the crisis, Beaumont appeared to be on the verge of economic stability after the 1982 oil recession. Traditionally a center for petrochemical and shipping industries, it has diversified gradually into service sectors; and while unemployment is still 13 percent, city leaders talk up their hopes of capturing the General Motors Saturn plant as an example of revitalized interest in southeast Texas.

"Five years ago, not every blue-haired old lady was worrying about where she could get the best interest rates," said Joe Sylvan, president of First City Bank, which holds most of the city's in-town investments. "All of a sudden, investments became a volatile kind of thing. All of a sudden, everybody's very rate-sensitive."

According to former mayor Maury Meyers, Beaumont invested all of its funds through local banks through 1982 . The city made its first investment with E.S.M. last May, a month after Finance Director Robert Nachlinger heard about the firm at a convention. He authorized loans totaling $62.5 million to E.S.M. over seven months, taking government securities as collateral.

These were so-called repurchase agreements in which E.S.M. bought back the securities after 60 days at a higher price, and Beaumont reaped profits of $857,000 on last year's deals.

The city made an additional investment of $20 million last January and February, and when E.S.M. failed, city officials found that they had no "safe-keeping" receipts for Beaumont's government securities supposedly being held by a New York bank. The bank, Bradford Trust, said it had no separate account for Beaumont's investment but acted only as a "clearinghouse" for securities investments.

Nachlinger, who resigned March 8, said in an interview last week that he wrote to Bradford twice last summer seeking the receipts. But he said the profits eased his mind.

"Son of a gun, it worked," Nachlinger said. "That kind of quelled any kind of fears I might have had."

Nollenberger, whose duties include acting as the city's finance director, contends he did not know that Nachlinger was investing with E.S.M. He called the investment an "error of judgment."

"How in the world could a city manager and city finance director send some securities off some place without getting some receipts?" Charles Tuttle, a Beaumont insurance salesman, said this week. "If they're going to do that, I'd like to talk to them, too."

"Nachlinger did some stupid things," like not getting the receipts, council member Sylvan agreed. "He just happened to pick the wrong horse to ride."

Among those who called for the state attorney general's investigation was a union local of the American Municipal, Federal, State and County Employes, which claims more than 600 of the city's 1,200 employes as members. Nollenberger is resisting their call for his resignation.

"He should have had this together," said Nick Toparcean, a city code inspector. "Our positions were jeopardized by these actions. How can we trust this kind of management in the future?"

"If there are any wrong-doers, we want them hung," union business manager Hugh Davis said. "If they're innocent, we want vindication. We want to remove the cloud hanging over Beaumont."