The nation's leading bankers said yesterday that they can't grant a request by President Reagan to lower interest rates, and they told him that the rates will drop only if he and Congress agree to a "sensible" budget compromise.

Llewellyn Jenkins, president of the American Bankers Association, predicted that the prime lending rate, currently 16 1/2 percent, will fall four percentage points by the end of the year if the administration and Congress agree to a plan for reducing budget deficits. But so long as Congress and the administration remain stalemated, investors' fears of a new round of inflation will keep interest rates high, he said.

Reagan told them high interest rates are standing in the way of economic recovery and he believes that, with inflation lessening, interest rates also should be falling. Aides said the president told the 22 bankers that some downward movement in the rates could go a long way toward ensuring prompt passage of the budget.

Jenkins, vice chairman of Manufacturers Hanover Trust Co. of New York, the nation's fourth-largest bank, later denied that the president had attempted to pressure the bankers into lowering interest rates. Earlier in the day, Jenkins said that the 22 bankers who met with Reagan yesterday indicated to him that they are willing to lobby for a budget package similar to the one proposed by the Republican-controlled Senate Budget Committee and endorsed by Reagan last week.

"He's got his eye on the right target," Jenkins said of the president. "Now, how he and the Congress are going to work with it, don't ask me that, because I don't have that knowledge."

Since Reagan embraced the Senate Budget Committee plan, GOP support for it has unraveled, with House Republicans rejecting the controversial proposal for cutting $40 billion from Social Security over the next three years. The White House has continued to support Senate passage of the proposal fashioned by Sen. Pete V. Domenici (R-N. Mex.), but spokesmen said on Wednesday the president would not insist that Social Security cuts be included in any budget bill passed by the House. Domenici explained his proposal to the bankers at the White House meeting yesterday.

"If anybody said they were asking us here to lower interest rates, they didn't quite understand how interest rates are handled because banks do not raise or lower interest rates depending upon how they feel about it," Jenkins said.