The Social Security rescue bill was stalled in the Senate yesterday by the second effort in two weeks to block withholding of taxes on interest and dividends.

The Senate refused, 58 to 37, to table an amendment by Sen. John Melcher (D-Mont.) that would delay the effective date of the controversial provision for six months, until next Jan. 1.

Finance Committee Chairman Robert J. Dole (R-Kan.), who angrily opposed Melcher, said the withholding provision would raise $20 billion over the next five years by catching income that now escapes the tax collector. But the banking industry wants withholding killed.

After losing the vote to table the Melcher amendment, Dole blocked a final vote on it, leaving the Senate at an impasse on the amendment and the Social Security bill.

The debate was a virtual replay of one last week, when a $5.2 billion anti-recession jobs bill was held up for several days in a dispute over an amendment by Robert W. Kasten Jr. (R-Wis.) to repeal the withholding provision. Kasten finally agreed to leave his proposal off the jobs bill after Senate leaders promised that it would be debated as part of a relatively minor trade reciprocity measure on April 15, income tax filing day.

House-Senate conferees agreed last night on a compromise jobs bill of about $4.6 billion. The compromise contains funds to enable 27 threatened states and the District of Columbia to keep paying unemployment benefits.

Dole, who said he would filibuster against the Melcher amendment until it is dropped from the Social Security bill, sharply attacked "the banking lobby."

After the tabling motion failed, he sarcastically offered an amendment to postpone the effective date of the withholding provision until Jan. 1, but only if the prime rate charged by the nation's 10 leading banks as of June 30 is 6 percent or less and stays that way. It is now 10.5 percent.

Dole lost, 57 to 35.

Dole accused the banking industry of stirring up opposition to withholding by misrepresenting it as a new tax when it is only a new method of collecting those already owed.

Melcher said he believes estimates of how much the Treasury will realize from the provision, which requires banks to withhold 10 percent on interest and dividend payments from most recipients, were "over- estimated."

The Social Security bill has been labeled an emergency measure by President Reagan and the leaders of both parties in Congress. Without some effort to shore up the system, the old-age trust fund would be unable to pay benefits after July 1. A bipartisan presidential commission worked out a $165 billion rescue plan which cleared the House March 9 by a 282-to-148 vote.

The bill sent to the Senate floor by the Finance Committee would raise the basic retirement age to 66 in the next century and cut basic retirement benefits then to 5 percent below the level of current law.

More immediately it would postpone this year's cost-of-living increase in benefits from July 1 to next Jan. 1; accelerate already scheduled Social Security tax increases so that the rate would be 7 percent each on employers and employes on the first $37,800 of earnings in 1984, instead of 6.7 percent; tax half the benefits of better-off retirees, and bring new federal workers into Social Security starting next year.

It also would raise Social Security taxes on the self-employed and remove the $6,600 annual earnings limitation for Social Security beneficaries by 1994.

Additional provisions would fundamentally alter the way the government pays Medicare bills by instituting a prospective payment system setting fixed rates in advance for hospital services to Medicare beneficiaries, and extend unemployment benefits for the long-term jobless. Some who have exhausted the current maximum of 55 weeks of benefits could get up to eight more.