The Senate yesterday returned to the controversy over withholding of taxes from dividends and interest, with a critical test vote likely to be taken next week.

As an estimated 15 million taxpayers were meeting the deadline for filing individual tax returns at midnight yesterday, Sen. Robert W. Kasten Jr. (R-Wis.) moved to repeal the withholding provision enacted last year and immediately filed a petition to cut off debate.

A vote on that cloture petition must occur on Tuesday, and it will be the first gauge of whether those who favor repeal have the votes to override an expected presidential veto.

The Senate postponed debate on the issue as some Republican members sought to work out a compromise and prevent a split in the party, but the deal was quickly rejected.

"I don't see any reason to compromise," Sen. Robert J. Dole (R-Kan.), chairman of the Senate Finance Committee and a leading proponent of withholding, said. He was promptly backed up by Majority Leader Howard H. Baker Jr. (R-Tenn.). Withholding of taxes on dividends and interest will bring the Treasury an estimated $13.4 billion over five years.

Kasten, backed by the anti-withholding banking industry, claimed to have the 60 votes needed to limit debate, but said it may take him two or three tries.

But Dole and a small band of supporters are preparing to filibuster, and they intend to continue stalling even if the Senate votes to limit debate.

Under Senate rules, it is possible to conduct what is known as a "post-cloture filibuster" by demanding repeated roll-call votes on secondary amendments. Dole yesterday filed 513 such amendments; 229 would change the Kasten proposal and the remaining 284 would change the minor trade bill to which the Kasten is seeking to attach his amendment.

The compromise rejected by Dole and Baker would have repealed withholding, but it also would have increased penalties for failure to report dividend and interest income and beefed up the Internal Revenue Service's monitoring system.

In addition, it would have required withholding for anyone found to have failed to report dividend and interest income. It may still be offered as a modification of the Kasten proposal.

In other tax matters, the IRS announced that the interest penalty for those who file late returns will drop from 16 to 11 percent on July 1.

Taxpayers had until midnight last night to request a delay in filing their taxes. Persons who request a delay also will be allowed up to a four-month extension to set up an Individual Retirement Account (IRA) for up to $2,000, and deduct that from their 1982 taxable income, the IRS said.

Meanwhile, the Senate Budget Committee remained deadlocked on taxes for the next fiscal year. Dole indicated he would be satisfied if the committee did nothing more than go along with President Reagan, who would hold the line against big tax increases in fiscal 1984 and 1985 but allow $150 billion in increases over the following three years.

As behind-the-scenes bargaining continued yesterday, sources said Democrats were insisting on at least $15 billion in new taxes next year, while Budget Committee Chairman Pete V. Domenici (R-N.M.) won't "go into double digits," as one Republican aide put it.