Sen. Frank Church (D-Idaho) yesterday rejected President Carter's compromise plan for selling warplanes to the Middle East, setting the stage for a floor fight in the Senate next week.

The White House spelled out its compromise plan just two hours before Church told reporters it was not sufficient. The plan involves selling Israel 20 more F15 fighter-bombers in 1983-84, and giving Congress assurances that Saudi Arabia will not use the 60 F15s it hopes to acquire as offensive weapons.

Church said this proposal "did not produce the kind of coalescene of support within the [Senate Foreign Relations] committee that would be necessary to avoid a serious confrontation or a bruising debate on the floor."

"Barring some new developments," Church said, he would vote today for a resolution disapproving Carter's plan to sell warplanes simultaneously to Egypt, Israel and Saudi Arabia. According to informed sources, Church's vote would give opponents of all three sales a 9-to-7 advantage in the Foreign Relations Committee.

Church has publicly spoken out against any plane sales to the Middle East at this time because of the delicate nature of the stalled peace talks between Egypt and Israel. But earlier this week he led administration officials to conclude that he would support Carter's compromise.

Yesterday, however, he decided to go the other way, in a shift that angered some Senate colleagues. One key Democratic senator said privately that Church had waivered inexcusably.

Another Democrat more friendly to Church said he never heard the Idaho Democrat - who is due to become chairman of Foreign Relations next year - explicitly endorse the Carter compromise.

At a meeting with Secretary of State Cyrus R. Vance Tuesday, Church asked if the administration would consider adding to the 75 F16 fighter-bombers (a weapon primarily intended for offensive purposes) it propose to sell Israel. Vance rejected this idea, sources said.

The upshot of Church's announcement was to leave the White House where it had earlier expected to be - with the Foreign Relations Committee aligned against its proposed plane sales. Under the Foreign Arms Sales Act, the resolution of disapproval the committee is now expected to adopt today is a privileged motion that can come almost immediately to the floor. After a limited debate the Senate will vote on each of the proposed sales independently.

Carter formally offered the compromise in a statement issued by White House press secretary Jody Powell.

Powell said the president is prepared to sent to Congress next year a proposal to sell 20 more F15s to Israel, with delivery of the planes scheduled for 1983-84.

Cupled with the 25 F15s already committed to Israel and the 15 F15s that are part of the pending package, that would give Israel 60 of the sophisticated fighters by the mid-1980s - the same number the administration proposed to sell to the Saudis.

Powell said Carter has also promised to give "sympathetic consideration" to other Israeli military requests beyond the 20 extra F15s.

In addition, Powell said, the administration will send to Congress "specific written assurances" placing limiting conditions on the proposed sale of 60 F15s to Saudi Arabia in the pending warplanes package. Those assurances, in the form of a letter from Defense Secretary Harold Brown that is circulating in draft form on Capitol Hill, would include Saudi promises not to arm the F15s for offensive operations, base them close to Israeli territory or add to them with equipment or planes from other suppliers.

The administration made the offer over the last few days in a series of intensive negotiating sessions with congressional critics of the warplanes package sale. Powell said the offer has lessened the opposition, but he stopped short of predicting that a majority of the Senate Foreign Relations Committee and the House International Relations Committee will accept it and drop opposition to the pending sale.

Senate Majority Leader Robert C. Byrd (D-W.Va.), has counted 33 or 34 of the 62 Senate Democrats as supporters of the president's plan, according to informed sources. Republican senators are thought to be closely divided on the issue, with some of them reliably described as anxious to vote against the president in a bid for political and financial support from Jewish Americans who also oppose the package of sales.

The administration also hopes to win approval for the sales in the House, though this is unnecessary if the Senate approves them.

Earlier in the day, Powell denied a charge by Mark A. Siegel, a former White House aide, that Carter was giving in to "blackmail" by the Saudis in pushing the warplanes package.

Siegel, who resigned from the White House staff earlier this year to protest the warplanes sale, made the charge in a letter to members of the Foreign Relations Committee.

IN the letter, Siegel said that when the president visited Riyadh in January, Saudi King Khalid made it clear he wanted immediate action on the sale of the 60 F15s.

"The Saudi demand for immediate approval of the sale was tied to a discussion of OPEC (Organization of Petroleum Exporting Countries) oil prices and the falling value of the U.S. dollar aboard," Siegel said. "In other words, Khalid threatened the president of the United States with economic blackmail - and the president succumbed."

Powell noted that Siegel was not at the meeting, but that he and chief White House political adviser Hamilton Jordan were. He said that his and Jordan's recollections, along with a record of notes taken at the meeting, show that while the F15 sale was discussed by Carter and Khalid, oil prices and the dollar were not.

Powell called Siegel's allegaitions "total fabrications."

In a telephone interview last night, Siegel said he was told of the meeting by Jordan, who was his immediate White House superior, and that Jordan in discussing the meeting "made the connection" between the F15s and oil prices.

Siegel said if the notes show those subjects were not discussed "then obviously I misunderstood what was told to me or it was misrepresented."