Debate over whether to roll back the recent Social Security tax increase raged anew yesterday in the wake of Wednesday's House Democratic caucus vote calling for the use of income tax revenues to bolster the Social Security trust fund.

At a hearing before a Senate Finance subcommittee, former Social Security commissioner Robert M. Ball urged Congress not be panicked into undoing last year's payroll tax increase."It would be the wiser course not to legislate on this matter at all this year," he said.

At the same time, Rep. Al Ullman (D-Ore.), chairman of the House Ways and Means Committee, told a luncheon group that while he still personally opposes any reduction in Social Security taxes, he will not "ignore" the mandate of the caucus.

"I've kept my finger in the dike for a long time," said Ullman, who until now has been a major obstacle to rollback efforts momentum in Congress. "But there comes a point of diminishing returns . . . We may have to make some kind of adjustment."

The debate cames as Barry P. Bosworth, director of the Council on Wage and Price Stability, told senators that a $30 billion Social Security tax cut - as suggested by some law-makers - would result in reducing inflation by 2 percent over the next two years.

Although Bosworth reiterated the administration's official opposition to any Social Security tax changes this year, he nevertheless conceded the move would be a visible anti-inflation measure. The council chief has recommended it for the president's new wage-price program.

Opposition by Ball and Ullman was expected to have little impact on the rollback drive in Congress. If Wednesday's caucus is any indicator, the law-makers already are on the way toward voting some sort of payroll tax relief. The vote in caucus was 150 to 57.

Along with his comments on the payroll tax issue, Ullman made several other points at yesterday's lunchoen:

He predicated Congress would approve "a good part of" President Carter's new urban package. But he flatly ruled out enactment of a White House proposal to allow an extra 5 percent investment tax credit for companies that build new plants in poor neighborhoods.

He forecast that Carter would get about half of the $9.4 billion in "tax reforms" he has proposed, an assessment far more optimistic than most other congressional observers. Ullman also confirmed that Ways and Means will cut the size of the president's tax-reduction proposal.

He hinted that the lawmakers may move later to cut back on abuses by taxpayers in not declaring fringe benefits as income. The Ways and Means Committee has barred the Internal Revenue Service from cracking down on fringe benefit abuses, but may take up the issue itself.

Ball's testimony echoed the position of the Carter administration, which has asked the lawmakers not to act in haste to roll back Social Security taxes, lest they upset Carter's tax-reduction and "reform" package.

The White House proposed last year that the lawmakers use general income tax-revenues to "save" the Social Security system from bankruptcy over the short run. The two houses voted instead to raisepayroll taxes to restore the trust funds to solvency.