House Democrats and Senate Republicans yesterday set up major battles in both houses of Congress next week over U.S. policy toward rebels in Nicaragua, producing compromise proposals that largely are incompatible.

A senior administration official said President Reagan, in an effort to move some form of aid through Congress, was willing to provide only "non-lethal" aid -- anything short of weapons and ammunition -- to the "contra" insurgents fighting the leftist Sandinista government of Nicaragua, at least until Sept. 30, when the fiscal year ends.

This represented a major retreat from Reagan's repeated emotional demand in recent weeks that the United States express its resolve to combat Soviet influence in Central America with military support for the Nicaraguan "freedom fighters."

But it is not yet a victory for the Democratic view that the contras are waging an illegal, brutal and immoral war and deserve no funds.

In an independent effort to bridge the chasm, Senate Minority Leader Robert C. Byrd (D-W.Va.) organized a meeting of more than a dozen leading Democratic and Republican senators with Reagan Sunday at the White House to discuss further alternatives. Byrd declined to specify the proposals he will bring. The preliminary list of those attending spanned much of Senate opinion.

Byrd's move likely will affect all pending contra aid plans except for a proposal offered by House Democrats and three moderate-to-liberal House Republicans.

That plan, to go to the House floor next week, was drafted for the House leaders by Intelligence Committee Chairman Lee H. Hamilton (D-Ind.), Western Hemisphere Affairs subcommittee Chairman Michael D. Barnes (D-Md.) and Rep. James R. Jones (D-Okla.) It also bore the names of Republican Reps. Edwin V.W. Zschau (Calif.), Jim Leach (Iowa) and Willis D. Gradison Jr. (Ohio).

It would end further aid in any form to the contras and provide $10 million to the United Nations High Commissioner for Refugees for helping people "outside Nicaragua." Another $4 million would go to Mexico, Venezuela, Panama and Colombia to enforce any regional peace treaty that their Contadora peace process might produce.

Barnes ruled out any change that would send aid directly to the contras. Hamilton said the proposal does press Nicaragua. "If they do not cooperate toward a diplomatic solution, then they know they have strengthened the pressure for military aid for the contras," he said.

Secretary of State George P. Shultz denounced the House Democratic compromise effort as "unworkable" and "not an alternative at all" because it would "take all pressure off the Sandinistas." He said its provisions "virtually amount to a call to the rebels to surrender and leave the country."

House Minority Leader Robert H. Michel (R-Ill.) and other GOP leaders were drafting a plan to provide non-lethal aid through some government body other than the Central Intelligence Agency. But staff members said they would wait to see what emerged Sunday.

Sen. David F. Durenberger (R-Minn.) circulated a proposal with the administration's backing that would remove restrictions on the CIA "reserve for contingencies" fund and on the $14 million that Reagan sought earlier for military aid, "provided that none of the funds made available by this joint resolution may be used to provide arms, munitions or other weapons of war to any person, group or organization, directly or indirectly."

Several House Democrats expressed concern privately that the wording would allow unrestricted secret military spending from the CIA contingency fund, an interpretation that Senate Foreign Relations Committee Chairman Richard G. Lugar (R-Ind.) tentatively confirmed early yesterday.

But a senior administration official said the approach would not unlock the contingency fund, and Lugar later corrected his remarks. A senior administration official said, "If the president comes to an agreement with the Congress on release of the $14 million so that it will not be spent on lethal equipment, he would not use contingency funds for that purpose either."

Nevertheless, that draft also appeared dead. "I don't think this resolution is the one that will be at issue when we meet with the president," a Lugar aide said.