Senate leaders yesterday intensified efforts to renegotiate a budget compromise as President Reagan characterized an earlier plan as "totally unacceptable" and Senate Majority Leader Robert J. Dole said he could not vote for it.

In the House, Democratic leaders said several budget proposals are ready for possible action after Reagan and the Senate Republicans resolve their disputes over taxes, defense and other stumbling blocks.

But any optimism over a break in the budget deadlock was guarded as constraints from the Gramm-Rudman-Hollings budget-control law, coupled with election-year pressures, added to the normal complexities of producing a budget.

"There is no compromise that I'm aware of yet, and I'm not sure that one will be [reached]," said Senate Budget Committee Chairman Pete V. Domenici (R-N.M.) after Republicans and Democrats huddled separately.

At issue is a bipartisan tax-and-spending plan proposed six weeks ago by the Senate Budget Committee. Reagan denounced that budget plan in a letter that was written earlier this week and released yesterday by Dole (R-Kan.) as part of his effort to keep the White House and Republican-controlled Senate from open warfare over the budget.

Reagan accused the committee of relying on "unacceptable tax increases and defense cuts" to meet the Gramm-Rudman-Hollings deficit target for fiscal 1987 and called on the Senate to rewrite the budget in line with his priorities.

Nearly half the Senate Republicans have expressed opposition to the plan for similar reasons, and Dole has been attempting to find common ground between them and Domenici without losing Democratic votes considered necessary. He has repeatedly said he wants a budget to pass with a majority of Republicans supporting it -- "not 35 Democrats and 18 Republicans," as he put it yesterday in what some senators say may be an accurate reflection of the current outlook for the committee plan.

Dole said yesterday that tax increases would have to be reduced to about $12 billion, halfway between what Reagan and the Budget Committee proposed. Other sources said defense spending increases of $5 billion to $6 billion are also under discussion, although the deficit impact for this year would be only about $2 billion. At issue are offsetting cuts in domestic programs, which most Democrats and some Republicans are reluctant to make.

The job may be made easier by what some senators are calling "found money," resulting from cost reestimates reflecting changing economic prospects, including a possible $3 billion savings from lower cost-of-living increases for Social Security and other pension programs.

Inflation is expected to fall below the 3 percent "trigger" for automatic cost-of-living increases, although Congress is expected to authorize an increase anyway. Still, it will probably fall well below the 3.4 percent increase allowed in the committee budget, providing a substantial bookkeeping savings.

On the Senate floor, a proposal from Sens. Dennis DeConcini (D-Ariz.) and James Abdnor (R-S.D.) to transfer $168 million from foreign aid to law enforcement programs was approved.

Republican leaders agreed with Sen. Edward M. Kennedy (D-Mass.) to try in separate legislation to limit an increase in hospitalization costs paid by Medicare recipients. The deductible of $492 is scheduled to rise to $572 next year; the Republicans agreed to try to limit it to $520.

The Senate also rejected, 65 to 32, a proposal from Sens. Gary Hart (D-Colo.) and Lawton Chiles (D-Fla.) to raise taxes by $3 billion for "investment initiatives" in research, resource development, education and training.

House Majority Leader James C. Wright Jr. (D-Tex.) told reporters at a breakfast meeting yesterday that one of the budget options being considered by the House leadership would meet the $144 billion deficit target without raising taxes by imposing a series of cuts and freezes, roughly half on defense spending and half on domestic programs.

Asked if Reagan would like this proposal, Wright said, "No, I don't think so. He hasn't liked any budget resolution we have passed."

Wright defended the continuing refusal of House leaders to consider tax increases to reduce the deficit unless such a step is first endorsed by Reagan and supported by GOP leaders in both chambers.

"So long as the president is adamantly, persistently maintaining that he will veto any tax bill, what the hell do you think we're going to do?" he asked. "Do you think there is a two-thirds vote in the Senate to override a veto?"

Wright said that if the Senate demonstrates after "many, many weeks" that it cannot produce a budget, the House would act on its own.

House leaders have long argued that budget cuts made by Congress this year are likely to fall evenly between defense spending and domestic programs because there is little incentive to deviate from the deficit-reduction formula built into Gramm-Rudman-Hollings that would be triggered if Congress fails to produce a budget.

House Budget Committee Chairman William H. Gray III (D-Pa.) said the proposal described by Wright is one of the options being considered by House leaders.

Gray said House action on the budget "all depends on what happens in the debate between [Senate and administration] Republicans over methodology -- more revenues, budget cuts or both." He said Reagan is "boxing himself into a corner" by refusing to negotiate on revenue increases. "The stuff he cares about is going to get savaged," Gray said.