The Brookings Institution, in a study published today, challenges supply-sider claims that federal tax cuts will pay for themselves by boosting economic growth.

Supporters of the administration's proposed 30 percent, three-year across-the-board tax cuts say that reducing individual tax rates will have a measurable effect on incentives to work and to save.

The new study, which reports the findings of a conference held in 1979 on how taxes affect economic behavior, concludes that, while the tax cuts would have some effect on how long or hard people work, they would not raise output sufficiently to offset the cost to the treasury, "even with the most generous estimates of the total labor supply response."

Many experts believe the administration is over-optimistic about its economic assumptions, and they dispute the claims that supply-siders make for the effects of tax cuts. But there is not very much empirical work on the subject.

The Brookings study brings together papers by a dozen economists working on how taxes affect business investment, corporate financial policy, the stock market, capital gains, housing investment and prices, saving and charitable contributions as well as work effort.

It concludes that tax "reform" to get rid of distortions caused by inflation is at least as important as tax cuts top improve the economy.

Inflation has a big impact on how taxes affect the economy, the study said. Treasury figures published recently show that inflation would wipe out much of the tax cut proposed by the administration for a family with a median income. These use the official inflation forecasts.

Nevertheless, the administration claims that if Congress enacts President Reagan's tax cuts for individuals, there will be a big rise in work and saving.

One of the Brookings papers suggests that the willingness of men to work may be more affected by taxes than previously thought, although married women still respond more to tax changes than do men.

On the other hand, there is inconclusive evidence of the effect of tax cuts on saving, the study says. "The prevailing view is that saving is somewhat responsive to changes in the rate of return," the summary says, "but we don't know by how much." It concludes that "it is difficult to change national saving by government policy."

Business investment is significantly affected by changes in tax rates, investment write-offs and tax credits, another study concludes. However, the demand for business products and the level of interest rates have an important impact as well, experts say.

Capital gains tax rates do not have much effect on capital realizations overall, a paper in the volume says. But if only large portfolios are considered, a rise of between 11 percent and 20 percent in realizations could result from a drop of 25 percent in the capital gains rate.

Some economists argue that a lower tax rate on capital gains would encourage people to sell their investments more frequently rather than keep capital gains "locked in" to avoid taxes. This would boost the stock market and increase efficiency, they say.

The administration does not propose cutting the capital gains tax directly, but cuts in the tax rate in the top bracket will reduce the maximum tax payable on capital gains.