Within the next year, Congress and President Reagan must face one of the most serious domestic problems on the national agenda: How to rescue the giant Social Security system from a financial shortfall that could shut off benefits in the next 18 months.

With Republicans and Democrats currently locked into irreconcilable positions, the legislative struggle could dominate the next Congress and leave political victims from one end of Pennsylvania Avenue to the other.

Reagan, who recommended deep benefit cuts in 1981 as a solution, was repudiated even by many leaders of his own party and had to back off and appoint the special national study commission that is due to start voting on recommendations Nov. 11.

The commission, headed by Alan Greenspan, former chairman of the Council of Economic Advisers under President Ford, has 10 members appointed by the Republicans and five by the Democrats. If its members could agree on a solution, that could be a compromise that bridges the gap between the White House and congressional Democrats and leads to swift legislative action. But its findings are not binding and recommendations from similar commissions have been largely ignored in the past.

Social Security, with projected outlays of $229 billion for old-age, disability and Medicare payments in fiscal 1983 (including the $18 billion portion of Medicare paid directly from the general revenues of the Treasury rather than the trust funds) is by far the nation's largest retirement and disability system. And it also is a major portion of the federal budget, accounting for nearly 30 percent of all outlays in fiscal 1983.

Nearly 36 million people draw monthly benefits. The maximum benefit for a person retiring at 65 this year is $729 a month, and if he or she has an aged dependent spouse an extra 50 percent is paid, for a total of $1,093. The average benefit for a retired worker is about $416 a month. Medicare covers about 75 percent of hospital costs for the aged.

The immense role played by Social Security in the lives of the aged is shown by this figure, based on government surveys: Two-thirds of the aged get at least half their cash income from Social Security.

With people depending that much on Social Security for living expenses and on Medicare for paying hospital bills, it isn't surprising that Social Security, created in 1935 as part of Franklin D. Roosevelt's New Deal, has become one of the most popular government institutions. Attempts to cut it are deeply feared and fiercely resented by an ever-growing and politically weighty segment of the population.

For two generations Social Security rolled along with few financial problems: The biggest issue was how to boost benefits and make them more adequate.

Now, however, the system is in serious trouble. The old-age trust fund is already operating at a deficit and will be able to get through to next July only by borrowing from Medicare, which is temporarily flush but will face its own problems a bit further down the road. If additional borrowing is authorized by Congress beyond July, the system can probably get through to mid-1984.

The basic reason for the immediate problems of the old-age fund is the extremely bad performance of the U.S. economy over the past five years. In 1977, when the last major Social Security bill came up, Congress looked at the system and, assuming the economy would perform at a middling, reasonably successful rate (called the "intermediate scenario"), approved a new schedule of taxes intended to make it sound.

It was confidently assumed then that the tax schedule would fund the old-age and disability systems comfortably well into the next century. This year that schedule calls for 6.7 percent each from wage earners and their employers on the first $32,000 of earnings but will rise to 7.65 percent by 1990. (For self-employed persons the tax is 9.35 percent this year and will rise to 10.75 percent by 1990.)

However, the economy hasn't performed as well as expected. It has even been worse than the pessimistic scenario. In recent years inflation has outpaced the growth of wages. As a result, the payroll tax has brought in relatively less than anticipated, and the annual automatic benefit increases pegged to the consumer price index have forced the system to pay out relatively more than anticipated. The result: a serious shortfall of money.

But the shortfall isn't permanent. If the economy should return to the intermediate scenario, then the old-age fund, once through the current crisis, would start to build up again toward the end of the decade and would be in good shape until about the year 2025.

At that point there would be a second problem: The baby boom generation of the post-World War II era would reach retirement age and the trust funds under present tax schedules would rapidly be depleted.

So the commission and the government are really faced with the two problems on the old-age and disability funds: the immediate shortfall and the long-term problem caused by the aging of the population, which puts more strain on the system.

For the short-run problem, there really are only a few alternatives available for the commission, Congress and the president. Massive cuts in the benefits of those already on the Social Security rolls or just about to retire are politically unpalatable, except perhaps for a trim in the annual cost-of-living adjustment (COLA), perhaps to three-quarters or four-fifths of the rate of the CPI.

The short-run options seem to be some combination of the following: borrowing from Medicare for another year or so (but no longer, since Medicare itself will be in bad trouble soon); obtaining a line of credit from the Treasury (or alternatively making grants from the Treasury although the Reagan administration and many in Congress are dead set against this); moving up to 1983 and 1984 the Social Security payroll tax increases now scheduled to occur in 1985, 1986 and 1990; bringing in federal, state and local government employes to provide a short-run rush of new revenue; and some benefit trims, such as in the annual COLA.

The bulk of congressional Democrats seem strongly opposed to benefit cuts, prefer the idea of general revenue funding or Treasury borrowing and probably could support moving up the rate increases. Republicans dislike the idea of using general revenues or borrowing from the Treasury. They might go along with moving up the tax and substituting selected trims for Treasury grants or loans.

There are also disputes over what to do about Social Security in the long run. A tax increase of 1 percent each on employers and employes beyond the 7.65 percent level could carry the system through to about 2060, and many Democrats appear to be leaning to some such solution, or to long-term general revenue financing.

Many conservatives, however, favor phasing in some long-range permanent trims in benefits to reduce the future burden of the system and thus avert the need for major tax increases.

One approach involves phasing in -- starting in the 1990s or around the turn of the century -- an increase in the basic age for retirement without loss of benefits (now 65), perhaps to 67 or 68. People live longer now, it's argued, and can keep working longer.

A reduction in the basic benefits paid to new retirees (perhaps 10 percent) could also be phased in it is argued.

However, Democratic spokesmen, such as former Society Security Commissioner Robert Ball, contend that long-term predictions of disaster may prove wrong and that it would be foolish to cut benefits to meet a crisis that may never happen.

All these projections assume the intermediate scenario. If, instead, the pessimistic economic scenario proves accurate in the long run, the Social Security system's problems would be immensely worse. The current temporary shortfall would become permanent. And a payroll tax increase of about 3.25 percent each on employers and employes above the 7.65 percent rate would be required to pay for benefits through the next 75 years.

Those are the problems of the retirement system. As for Medicare, though it has a good balance now and its problems are a few years further away, it is actually in much worse shape. Under the pessimistic economic scenario it would go permanently bankrupt in 1988; under the intermediate the bankruptcy would occur in about 1991; and if times are good (which few expect), it could hang on until about 1995.

The basic reason Medicare is headed for bankruptcy is the sharp rise in hospital and medical costs, which have far outstripped the general cost of living rise year after year. Unless these costs are contained, Medicare cannot keep going.

One possible solution: Instead of the current reimbursement system, which is essentially a cost-plus mechanism, substitute a prospective payment system, in which fees for each type of patient are fixed in advance on a per-case basis. Secretary of Health and Human Services Richard S. Schweiker is already drafting such a plan at Congress' instructions.

Other proposals: increased cost-sharing by the patient in doctor and hospital bills; higher premiums for the doctor-care (Part B) insurance portion of Medicare for which the premium now is about one-quarter of the total cost (Treasury pays the rest out of general revenues); means testing for eligibility for the program; and use of some general Treasury revenues.

The administration also is soon to propose reorganization of the whole medical reimbursement system in the United States to discourage the frivolous use of health facilities allegedly spurred by too generous tax deductions for health insurance premiums, and to encourage insurers to compete for business by lowering costs and making hospitals get more efficient.

Medicare is now on a back burner. But as soon as the more immediately pressing problems of the old-age system are dealt with, the Medicare portion of the Social Security controversy will heat up. CAPTION: Picture, The average Social Security benefit for a retired worker is about $416 a month. By James M. Thresher -- The Washington Post