A majority of the House has brushed off the Democratic leadership and has signed a petition to force a floor vote on legislation repealing withholding of taxes on interest and dividend income.

The show of strength by the banking industry, which has lobbied to kill withholding, almost guarantees that Congress will repeal or significantly modify the provision, now scheduled to take effect July 1. The Senate has already passed legislation that would postpone withholding until 1987, if not forever.

After a day of closed-door sessions with Democratic colleagues, Rep. Dan Rostenkowski (D-Ill.), chairman of the Ways and Means Committee, appeared ready to throw in the towel and report repeal legislation directly to the floor as early as next Tuesday.

"I see no reason to continue to protect the president when a majority of the Republicans on both sides have walked from the president," Rostenkowski said. He said straight repeal legislation, as opposed to the Senate "compromise," would be the "honest way to move."

Proponents of repeal garnered the critical 218th signature on a so-called discharge petition before noon yesterday. House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) and Rostenkowski had been trying to defuse the petition drive.

Unless the Ways and Means Committee moves the legislation, the rules of the House provide that on May 23, the fourth Monday of the month, anyone who signed the petition can make a motion to force the House to take up the legislation immediately, removing it from Ways and Means' hands.

The leading advocate, Rep. Norman E. D'Amours (D-N.H.), declared that he would make such a motion if Ways and Means doesn't act. Before the petition gained 218 signatures, most members of the committee privately conceded that repeal was almost inevitable, but they were considering combining repeal with other tax increases, including new levies on banks, to make up the lost revenues.

Complete repeal would cost the Treasury a total of $13.4 billion through 1988. The Senate compromise legislation would result in the loss of at least $5.2 billion, as new reporting, penalty and auditing provisions would raise an estimated $8.2 billion.

Discharge petitions are rarely used in the House. Since 1933, when careful records began, majorities have signed such petitions only 36 times, and only once in defiance of the influential Ways and Means Committee. That was in February, 1933, forcing action on a tariff bill.

The gathering of 218 signatures is the latest development in an extraordinary demonstration of political leverage by the banking lobby. It has not only been able to defy the House and Senate leadership, but has run roughshod over President Reagan.

In an unexpected development, the American Bankers Association has been privately telling officials on Capitol Hill that the Senate bill, which was designed to eliminate the banks' withholding responsiblity, could prove more of a burden than full withholding.

The ABA objected particularly to a requirement that the Treasury compile a list of all persons who have either under-reported their income or not filed tax returns.

"It would be as long as the phone book for all five bouroughs of New York, and tellers would have to go through it" every time they opened up a new account or conducted a transaction involving interest payments, one ABA official complained.