The budget shortfall facing Maryland Gov. Harry Hughes next year has grown to more than $100 million because of a dramatic and unexpected increase in the cost of the state's pension system.

Hughes, who is contemplating raising state property taxes and holding new lotteries to overcome the 1984 budget gap, will officially get the bad news Thursday morning when the state's board of revenue estimates meets. But today financial experts (the actuaries who monitor the pension system for the state) told a legislative committee that earlier estimates for next year's cost of the state's retirement fund were $46 million low.

"That's a big chunk for the governor to come up with," said outgoing Senate President James Clark (D-Howard). "It means that social programs get squeezed, state employes' pay raises get squeezed. Everything gets squeezed because you only have so many dollars."

The new estimates on what it will cost to keep the pension system financially sound came as more bad news for Hughes, who earlier had been told that his fiscal 1984 budget, which by law must be balanced, was short by more than $61 million. State officials blame the shortfall on the depressed national economy, which has kept down state sales tax collections and other revenue sources.

Hughes, who learned of the new pension fund estimates in the last 10 days, said he is waiting for Thursday's revenue estimates and discussions with state legislative leaders before making a decision on how to offset the shortfall.

In addition to raising the property tax a penny or two above the current 21 cents per $100 of assessed value and instituting new lotteries, the governor is considering a variety of actions to overcome the projected deficit. Administration officials said the options include juggling federal aid and possibly further reducing budget requests from departments.

Hughes has said that because of the budget problems, he may not be able to include state employe raises, targeted education aid, increases in welfare benefits or new initiatives in the fiscal 1984 budget he will send to the legislature after it convenes in January.

"This aggravates the budget problem," said Ejner J. Johnson, Hughes' chief of staff. "Everything is compounding to make the whole picture bleaker than before. Obviously we have to increase the number of options. The governor is going to want to talk to the leadership before he makes any decisions." Johnson said that Hughes is not considering raising the state's sales tax.

Until two weeks ago, state budget officials expected next year's contribution to the pension system would be $386.6 million, the figure given by the state's former actuarial consultants. However, a newly hired actuarial firm reviewed the figures, using more up-to-date statistics, and found that the state would have to pay $432.2 million in the 1984 budget.

By law, Hughes is forced to accept the figure provided by the actuary.

Three years ago the legislature revamped the pension system because of spiraling costs caused in part by a cost-of-living allowance that was tied to the consumer price index. The new law caps the cost-of-living at 3 percent for new employes. Most state retirees and employes, however, are still covered by the old law.