The nation's nagging Social Security problem is back again.

Barely a year ago, it seemed Congress finally had found an answer for the ailing Social Security system: A sharp jump in payroll taxes to save the trust fund from going broke.To be sure, the higher taxes might prove uncomfortable, but that was 13 months away. Meanwhile, bankruptcy fears would be assuaged.

Now, with those payroll tax increases scheduled to take effect in January, there's talk of Congress reconsidering some of its 1977 actions. The House Ways and Means Committee already has pledged to "look into" the Social Security issue again next session, possibly to head off further hikes in 1980.

Even more sweepingly, the Carter administration is exploring the possibility of proposing a thorough overhaul of the entire Social Security system next January, including everything from how to finance the system over the longer-run to who should be covered and how big benefits should be.

In short, the whole issue may go back to square one.

What's precipitated all the scurryingis the sudden realization that the really sharp tax hikes voted in late 1977 are just around the corner - making it likely there'll be a voter protest as soon as Congress returns in January. (A similar backlash last spring fizzled by summer, but the tax bite was less.)

This year, the maximum increase was only $105.60 a worker - high be historical standards, but still bearable enough once the initial shock wore off. But next year's will be $332.82 a person, making the pain - and the protest - likely to linger longer. (The boost will take effect on January 1.)

The increases stem in part from a rise in the payroll tax rate for employes to 6.15 per cent, from the present 6.05 per cent. But the real jump this time affect those in the upper-middle income brackets. The total wages subject to Social Security taxes will rise to $22,900, from $17,700 now.

The changes will bring the maximum payroll tax for a high-income worker in 1979 to $1,403.77, compared to $1,070.85 this year. In 1979, the maximum amount any wage-and-salary worker had to pay in Social Security taxes was a relatively small $374.40. Under today's law, by 1981 the maximum will hit $1,975.05.

The major impetus next year most likely will come from the administration's proposals, which are expected to beunveiled in President Carter's January economic message. A joint Tresuary-Health, Education and Welfare Department task force is developing the proposals now - although its work is going slowly.

If policymakers go ahead with all the options, the plan could call for everything from requiring federal workers to participate in the Social Security plan to financing all but the old-age retirement program with general income-tax revenues.

Final decisions haven't been made yet, but officials said the plan almost certainly will be more sweeping than the package Carter proposed in 1977. That plan, which sought to use some general revenues to finance the disability insurance portion of the present program, was rejected flatly by Congress.

The issues the administration is considering include:

GENERAL REVENUES: Should Carter try again to propose using general income-tax revenues to finance those portions of the Social Security program that do not relate to the basic old-age subsistence program - specifically, disability insurance and hospital insurance under the Medicare program?

The consensus among Social Security experts is the move would be one of the simplest ways of restoring fiscal integrity to the trust funds without "breaking into" the link most voters still believe exists between the payroll tax and retirement benefits.

It also would place the burden of financing these social programs back on the federal income tax, which is less "regressive" than the payroll tax. The payroll tax hits the poor proportionally harder than high-income taxpayers. But the income-tax is graduated, placing a larger burden on the rich.

The problem is, Congress is under the pressure from elderly persons not to place Medicare hospital insurance under general revenues, in part because voters like the perceived "protection" of the trust fund, and in part because they don't want their benefits to seem like "welfare."

Rep. Dan Rostenkowski (D-Ill.), chairman of the House Ways and Means subcommittee, repeatedly has blocked the shift on both those grounds. "My people don't want to think of this as welfare," he once told a hearing on the issue. "It's their right under the (Medicare) program."

COVERAGE: Should Congress require federal workers to participate in the Social Security program, as a way to help keep it financially sound? Federal workers now are exempt from Social Security taxes, on grounds that their federal pensions serve as a substitute forretirement benefits.

The measure would be easy enough to enact, but has influential opponents who so far have managed to keep it off the books. However, several Social Security study groups have recommended the move, and it's likely to be looked at closely this time around as well.

BENEFITS: Should Congress trim back future Social Security benefits - and tax rates - by easing present requirements that both grow in line with inflation. The escalator clauses were enacted when there was more room in the budget, but ultimately they've proved more costly than predicted.

Congress acted in 1977 to solve a portion of this problem by repealing earlier legislation that had boosted future benefit levels to reflect today's inflation. But critics say more cutbacks are going to be necessary if the system is to survive intact.

It's still too early to say precisely what finally will emerge from Congress next year. In the first place, the Carter administration itself hasn't yet decided what it wants to propose to the lawmakers, and probably won't until just before January's budget message.

And while the Ways and Means Committee may be interested in considering some shift to the use of general income-tax revenues for Social Security. Sen. Russell B. Long (D-La.), chairman of the Senate Finance Committee, is more likely to oppose it. The rest of the package will be up for grabs.

But some changes still seem likely. The odds are the lawmakers will make some effort to roll back the 1977 payrolltax increases - if only the assuage angryvoters who are likely to squeal over January's bite. And that means overhauling the rest of the system as well, if the problem is to go away.