The effort to find a compromise on a House plan to limit spending for franked mail collapsed yesterday after Democrats made a last-minute effort to increase the amount of money each member would be allowed to spend and Republicans refused to accept it, Republican and Democratic sources said.

As a result, the House Democratic leadership decided not to bring the $1.7 billion fiscal 1991 legislative branch appropriations bill, which contains the franking funds, up for a vote on the House floor before members depart for the August recess. "They will be working on a new compromise for September," a Democratic aide said.

Meanwhile, House members criticized a Senate measure approved Tuesday that would ban mass mailings to constituents during a year in which members are up for reelection. The proposal by Sen. Don Nickles (R-Okla.), offered as part of campaign finance revisions, would set rules for the House as well as the Senate. A House member who asked not to be identified said pointedly, "They set their rules, we will do our own."

House members who participated in yesterday's discussions on the frank -- the congressional privilege of using a signature in place of postage -- would not say how much more money for the taxpayer-financed mailings was proposed under the new Democratic formula.

Under a plan negotiated earlier by Rep. Vic Fazio (D-Calif.), chairman of the House Appropriations subcommittee on the legislative branch, and tentatively agreed to by a bipartisan group, each member would get about $160,000 annually for mail and would be required to file disclosure reports on how much was spent.

The proposed $160,000 allocation, which would pay for first-class mail responses to constituents and four mass mailings to every household in a district each year, was apparently not considered adequate by some key Democrats, and this concern led to Fazio's unsuccessful attempt to increase the allowance, sources said.

As illustrated by the latest U.S. Postal Service report on House mailing for the three months ending in June, members are sharply increasing their spending on more costly first-class mail because they have been limited to three mass-mail newsletters to postal patrons in their districts.

Under the current rules, members can send unlimited first-class mailings and are not required to file disclosure reports.

Nickles's amendment, based on the current Senate system, would require House members to institute public reporting of their mass-mailed materials. It would require listing the cost of such mailings as well as the number of pieces sent on a quarterly basis.

The measure also would prohibit mass mailings by members of the House or Senate once the Postal Service determined that body had exceeded its appropriated funds for a year. The Postal Service previously told the House it does not want to be the enforcer of a congressional mail limitation program, but an aide to Nickles noted that the agency's stand stemmed from an earlier proposal that involved cutting off individual members of Congress who used up their mailing allowances rather than limiting privileges for the full House or Senate.

Nickles's plan would bar mass mailings during any calendar year when a member is a candidate, a provision that would prevent House members from sending out mass mailings every other year.

Nickles's proposal would also prohibit House and Senate members from supplementing their mail funds by transferring money from other office accounts or using personal or campaign funds. Senators now have that option -- a contentious issue with House members, who have no such right.

Fazio proposed allowing House members to transfer excess mail allotments to other office accounts as part of his proposal in an effort to win support from members who do not do much mailing.