The present upheaval in the nation's capital markets is likely to continue as a result of the conflict between the expansion of tax and budget policies and restrictive monetary policy being advocated by the Reagan administration, Brookings Institution economists warned today.

The administration's policy is based on optimistic assumptions, particularly about inflation, for which there is little support in experience, the economists say in their annual analysis of the federal budget. Brookings economist Barry P. Bosworth told reporters at a news conference this week that he expects interest rates to stay high on average, but with large fluctations, if the Reagan program goes ahead. Bosworth is one of five authors of the study.

Given the uncertainty about whether supply-side theories will work, "There is serious question whether the nation should risk tax cuts extending to fiscal years 1983 and 1984 until the magnitude of the spending cuts and the performance of the economy better known," the authors say. If real growth next year is 1 percentage point lower and inflation 1 percentage point higher than forecast, the fiscal 1982 deficit will be $14 billion higher, the authors say.

The administration is becoming concerned about rising interest rates and the apparent skepticism on financial markets about whether the president's economic program will reduce inflation. The White House is serching for additional spending cuts for this year to keep the budget deficit in line with projections.

Bosworth and Joseph A. Pechman, who also is a Brookings economist and a contributor to the report, said the "novel and controversial aspect of the Reagan program lies in the projection of a rapid decline of inflation in future years" with no direct restraint on wages and prices, and at a time of economic growth rather than slack. They suggest that such a decline is unlikely, and say that if it does not occur then the tax and spending policy of the administration is "clearly on a collision course with the monetary targets."

Reagan assumes that there will be a very sharp and sustained increase in the "velocity" of money, or the ratio of money to total spending in the economy. The Brookings authors point out that the rise "is outside the range of historical experience" and that, unless inflation does drop rapidly, "A rise in interest rates would be required if the monetary targets are to be met."

High interest rates hit investment hardest, and so the money restraint urged by the administration and promised by the Federal Reserve Board could "negate much of the effort to stimulate" investment, Bosworth and Pechman argue. Administration officials say that their tax and spending program will encourage saving and investment.

Although Reagan's program does not envisage a rise in unemployment to curb inflation, such as has happened in Britian under the Thatcher government, the authors conclude that if inflation does not come down as expected, "monetary policy and high interest rates will restrain rather than stimulate overall growth" and "the Reagan policies will not be very different in effect from" those of the Thatcher government.

The Brookings study was written before the president's new proposals for cutting Social Security benefits were announced. However, it includes a discussion of the problems facing the Social Security system and suggests that benefits should be partially taxable, the retirement age raised gradually, and the indexation of benefits altered, avoiding overcompensation for price changes.

The economists were largely critical of the president's Social Security proposals, saying that the penalty for those who retire early would hit lower-income, blue-collar workers, and that the suggests changes would be much more than needed to ensure the solvency of the system.

On other parts of the budget, the study concludes that "many of the economies President Reagan recommends are long overdue, though in a number of cases still quite modest."

Pechman commented to reporters that the president was inconsistent in his definition of the safety net -- the set of seven programs that he promised to leave intact to help the poorest and most needy, because he has proposed cuts, for example, in welfare programs, food stamps and low-income energy assistance, all of which help the poor.

Instead, the president could have gone for bigger increases in user charges, more cuts in water projects, cuts in tax expenditures such as the dedcuction for consumer interest, and could have been tougher on agriculture, Pechman said.

The Brookings authors believe that "modest reductions in effective tax rates may be possible during the next few years if Congress approves spending cuts of the size proposed by President Reagan, and if no new programs are added."