Trustees of the Social Security system warned Congress yesterday that the system's financial plight is even graver than previously thought, and the old-age trust fund will go broke next year unless action is taken immediately.

The report, released on the eve of Senate Finance Committee hearings on what to do about Social Security's financing problems, is expected to give momentum to Reagan administration demands for a major cutback in future benefits.

"This is bad news, not only for recipients but for taxpayers," said the committee chairman, Sen. Robert J. Dole (R-Kan.).

Although benefit cuts proposed by President Reagan on May 12 were criticized on Capitol Hill -- by Dole, among others -- as excessive, Congress knows there is a serious problem. House Social Security subcommittee chairman J.J. Pickle (D-Tex.) already has begun work on a bill.

A favorite package of many in Congress involves cutting back on the annual cost-of-living increase and eventually increasing to 68 the age at which a person can retire with full benefits. The Reagan administration, however, has been reluctant to endorse either idea.

In this year's annual report, the Social Security Administration used four different sets of population and economic assumptions to project the future of the system.

One is extremely optimistic, projecting very low inflation and unemployment and a sharp rise in real economic growth. The second, somewhat less optimistic, corresponds to the official Reagan projections. The third, a middle-of-the-road scenario, is somewhat less optimistic and corresponds with the one normally used as "most likely" in the past. The fourth is very pessimistic.

In addition, the report included a fifth "worst case" category for its projectons over the next five years only. Even the "worst case" assumptions, Social Security Commissioner John A. Svahn noted, still assume that the economy will perform better than it has over the last five years.

The trustees made clear that there are two Social Security financing problems. One is a shortfall of funds in the next few years in the old-age trust fund. Starting in 1982, the fund won't be taking in enough to pay all benefits. The basic reason for this is that the economy has performed more poorly over the last several years than anyone had expected when Social Security financing was last strengthened in 1977.

If the old-age fund were combined with the disability fund, which is solvent, the combined trusts would start picking up in the 1990s, once scheduled increases in payroll taxes had gone into effect. The combined trusts would then be solvent for roughly a quarter century.

The second problem, however, is long-range and permanent and will worsen after the turn of the century, when the post-World War II baby boom generation retires. With an aging populaton and relatively fewer workers to pay into the system, the three funds -- old-age, disability and Medicare -- will eventually begin running permanent combined deficits under all but the most optimistic assumptions.

Addressing the short-run problem, the Social Security trustees said that, regardless of how well the economy performs, by 1982 the old-age fund won't have enough money on hand to pay all benefits. The old-age fund, which is far bigger than the disability or Medicare funds, paid $100 billion last year in benefits to nearly 30 million aged persons, their dependents and survivors.

Even if the old-age fund were combined with the more affluent disability fund, which paid out $15 billion last year, the combined funds will fall short of needed amounts by late 1982. The most optimistic assumptions give the combined fund solvency only until early 1983.

And although Medicare will soon have a reserve of about $19 billion, that reserve is expected to begin declining in a few years. If all three funds were combined, the Medicare reserve could keep the other two funds afloat only until 1985 under the most likely projections.

"The fact is that despite the 1977 Social Security tax increase, the largest peacetime tax hike in history, the system still faces a short-term funding shortfall that could range from $10 billion to $111 billion over the next five years, depending on the economic assumptions used," Svahn said at a news conference.

Using the third set of economic assumptions, the "most likely" category, yesterday's report projects that the reserves of the combined old-age and disability funds will drop too low to guarantee timely payment of montly benefits by around the end of 1982 and stay that way until about 1996 or 1997. The reserves will gradually build up as new taxes begin to accumulate, but then the funds will become insolvent permanently between 2020 and 2025 as baby-boomers retire.

The long-range deficit of the old-age and disability funds over the next 75 years, using the most likely assumptions, is projected as 1.82 percent of taxable national payroll. That is a more pessimistic figure thatn was contained in last year's report, which projected a deficit of 1.52 percent of payroll.

That's just for old-age and disability. Medicare is an added problem. The hospital insurance trust fund, now flush, will start declining rapidly in a few years as hospital costs continue to escalate, and the situation would get much worse later on, as the proportion of aged increases. Under every scenario, Medicare will go broke soon -- the most optimistic assumptions give it until 1998; the most pesimistic until 1989.