A new Brookings Institution study predicted yesterday that President Carter's new fiscal 1979 federal budget will make a deficit "a certainty" in fiscal 1981 and leave "no room for" the second round of major-tax cuts Carter has hinted he may propose that year.

The report - the latest in the institution's series of annual budget reviews - says if present plans are followed, Carter is likely to end up with a $15 billion to $30 billion deficit in fiscal 1981, rather than the small surplus he has projected.

It said although a deficit that size "may be needed in 1981 to keep the economy growing," it would deprive the government of the funds that would be needed to finance a big tax cut or other initiatives, and would impede Carter's goal of holding spending to 21 percent of gross national product.

The projections underscore what many economists have cited as inherent conflicts among Carter's major budgetary goals.

The president has said he still stands behind his pledge to balance the budget by fiscal 1981 if the economy permits.

Carter already has ordered a major spending crackdown for fiscal 1980, designed to pare the federal budget deficit to 37.5 billion from $50.8 billion in fiscal 1979. Officials contend that would put the administration back on track down balance in 1981.

However, Brookings' analysis shows the only way the president can do that would be to make sizeable cuts in many existing programs while avoiding any big spending initiatives, such as the national health insurance bill he has promised to propose.

As a result, the administration will be forced to choose between keeping its promises for new spending programs or cutting taxes again in fiscal 1981 to compensate for the impact of inflation, the document implied. But the study made clear that the administration cannot do both.

The study also confirmed what critics have been saying for months about the president's new tax cut proposals - that, contrary to what the administration has implied, the reductions would not offset fully the effects of inflation and higher payroll taxes on upper-middle-income families.

The document showed Carter's tax plan would reduce the overall tax burden only for households with incomes below $15,000 a year. It said about 15 percent of all households actually would suffer sizeable tax increases - including almost half of those earning $30,000 or more.

The analysis was prepared before Carter's latest budgetary change - his decision to trim his $25 billion net tax cut proposal by $5 billion. However, Brookings economists said the shift would not affect their budget estimates dramatically, and would reduce the impact of the tax cut.

The 319-page document, entitled "Setting National Priorities," also made the following points, many of which have been advanced by critics of the administration in earlier statements:

Carter's new "zero-base budgeting" system has resulted in "virtually no" reductions in existing programs. The report said even the few new spending initiatives Carter has proposed were "decided outside" the usual budget process.

Although Carter has called for holding the growth of defense spending to 2.7 percent a year between now and 1983 after adjustment for inflation, Brookings projects that the rise in defense spending will exceed that target by $24 billion - implying larger budgets or bigger cuts ahead.

Although the administration's new urban policy is designed to reverse the flow of people and jobs from central cities, the report said it offers "no means of slowing the suburdan growth that has been depopulating large cities."

The president's last-minute proposal to boost federal college scholarship aid in order to head off enactment of a tuition tax credit in Congress was "hastily conceived and poorly designed," but has been improved by legislative committees and is now "superior to" tax credits.

Enactment of costly farm bills in 1977 and this year and will heighten the longstanding conflict between keeping farm incomes high and consumer food prices low. The report noted that "it remains to be seen" which way the administration will use the new legislation.

The document was the institute's ninth consecutive annual budget review.