President Reagan, facing a growing likelihood of a defeat on the Senate floor, yesterday failed to persuade key leaders of the banking lobby to back off the drive to repeal 10 percent withholding on dividend and interest income.

"We had a very pleasant and unproductive meeting with the president," William H. Kennedy, president of the American Bankers Association, said as he left a one-hour session at the White House. Kennedy claimed that the issue of repeal "really isn't up to us because the people of this country have made up their minds that this is a bad law."

The anti-withholding lobbying drive, one of the most intense in recent memory, won a major convert over the recent congressional recess: Rep. Robert H. Michel (R-Ill.), the House minority leader.

Before he left for the recess, Michel's position was " 'Let's try it for a year and see if it works,' but after 10 days in the district, he got a different message. He got hit everywhere he went," according to Michael S. Johnson, his press secretary.

On a separate Capitol Hill front, Sen. Jake Garn (R-Utah), chairman of the Banking Committee, sharply criticized the tactics used by the banks in the lobbying campaign, but added that still he would vote to repeal the withholding provision. The law is scheduled to go into effect July 1.

Garn told the Dealer Bank Association that the bank lobby campaign has distorted the issues and given the false impression that 10 percent withholding is a new tax when, in fact, it is a mechanism to improve collection of existing taxes.

On Friday, when the Senate takes up trade legislation, Sen. Robert Kasten (R-Wis.) has a commitment from the leadership that he will be given the floor to propose an amendment repealing the withholding provision. The first test vote, which is expected to be on a motion to cut off debate, may not be held, however, until the beginning of next week.

Almost all participants in the battle agree that there are strong majorities in both the House and Senate supporting the drive to repeal withholding. The real test is whether there are two-thirds of both branches willing to vote to override a probable presidential veto of any bill carrying repeal language.

At the White House session, Reagan, Treasury Secretary Donald T. Regan and key aides met with about 12 leaders of different banking groups. According to Treasury officials, the president told the bankers that "it is better to collect taxes than to raise them."