The anti-inflation yet antiregulation Reagan administration has a health cost problem sure to cause it a big pain in the budget.

Hospitals and doctors are faltering in a three-year-old attempt to control the nation's hospital bill, $98 billion a year and still mainly unrestrained.

As an alternative to Presdient Carter's effort to control hospital costs by federal edict, hospitals joined with the American Medical Association in early 1977 in an ambitious "V-E" or "Voluntary Effort" to control their spending. Its success helped them squash Carter's legislation.

In 1980, however, hospital spending grew at a rate around 25 percent greater than general inflation. Hospitals are treating more people, buying more new equipment and doing more and more surgery, no matter all the programs in place to cool these trends. Hospital costs are the largeest, fastest-growing part of a national health bill running $243 billion a year, well above defense spending.

The new administration, dismayed by rising federal spending for just one health program, Medicaid, is talking of cutting Medicaid allotments to the states. "This will just shift some costs to local governments," said a politically conservative hospital official and Reagan supporter. "It won't touch their effect on inflation."

Leanding health officials have been warning their constituencies that an anti-regulation administration may still regulate.

Alex McMahon, president of the American Hospital Association, blames the increased spending mainly on a sharp and unexpected growth in hospital use especially by the aging, on economywide inflation, and on higher wages for more employes needed to care for more patients. Other authorities find other reasons -- the nursing shortage (triggering a "nurse price war" between hosptials, with a ripple effect on other wages); empty beds; hospital "marketing" (aggressive sales methods to attract new patients and doctors); unneeded admissions, tests and medications by doctors; and the increase in costly surgery.

Whatever the reasons, McMahon has warned, if hosptial spending and hiring continue, they may well cause "cost increases unacceptable to even the most understanding legislators and business executives." Michael Bromberg, executive director of the Federation of American Hospitals, the investor-owned hospitals, thinks President Reagan is going to be "every bit as tough" as Carter on hospitals. Dr. James Sammons, AMA executive vice president, warns that if Reagan "means what he says" about the budget and tax relief, "something has to give." If doctors fail to restrain fees and hospital use, Sammons says, medical care will be forcibly "downgraded" by a hard-pressed government.

Hospital spending is already devastating Medicare and Medicaid budgets and beginning to trigger drastic responses. Maryland, for example, has put a "temporary" 20-day limit on Medicaid payment for hospital care, but asked hospitals not to expel the patients. This means the unpaid bills must somehow be paid by other patients.

Industry, seeing a 12 to 30 percent annual boost in its health insurance bills, is talking of more employe "cost-sharing," that is, less insurance and more cash out of the pocket for sick workers and families.

No one is sure why hospital use has been growing. In part, it may be a "recession effect," with more use by workers either just laid off or worried about layoff. For whatever reasons, community (nonpublic) hospitals were 76.6 percent filled on average in the first half of 1980, their highest occupancy in nine years. Third-quarter admissions were up 2.7 percent; admissions were up 2.7 percent; admissions of patients over 65, consumers of three times as many services as younger patients, an astonishing and unexplained 7 percent over 1979.

Hospital spending had grown by a comparatively restrained 13.1 percent in 1979. The V-E's 1980 goal was an 11.8 percent increase, "contingent on" no increase in general inflation. From January through September, community hospitals instead spent $58.2 billion, up 16.3 percent from the like months in 1979. September spending alone as up 19.2 percent over September 1979.

Health economist Paul Ginsberg of the Congressional Budget Office thinks the V-E perhaps "trimmed all the fat," so it was harder to trim further. Robert Helms of the American Enterprise Institute says: "To an economist, the V-E is just "arm-twisting" and in the long run "any constraint program is a losing battle" when "it's in each hospital administrator's interest" to collect more -- to stay in the black, to modernize, to compete.

A federal analysis indicates that 53 percent of increased hospital spending from 1969 to 1979 was caused either by general inflation or by superinflation in hospitals' market baskets, and 18.7 percent by population growth and more use. This still left 21.9 percent caused by increased "intensity": ever more technological and intensive care, some of it clearly life-saving, some of it unproved or far from proved.