Members of the House and Senate banking committees, bowing to pressure from the White House yesterday, agreed to give $10.8 billion to the insolvent S&L insurance fund and to allow securities firms and "nonbank banks" to buy failed S&Ls.

Lawmakers reached an agreement by reopening a conference committee, which three weeks ago had drafted a compromise banking bill. That bill would have provided only $8.5 billion to the Federal Savings and Loan Insurance Corp.

The unusual parliamentary move, which some congressmen described as extraordinary, was taken to avoid a threatened veto of the earlier banking bill by the White House. The administration argued the previous banking bill was anticompetitive and failed to provide enough money for the insurance fund, which has been drained by a record number of failed S&L in recent years.

By reopening the conference committee to change the banking bill, the chairmen of House and Senate banking committees insured that the full Senate would not be able to amend the bill, according to congressional aides. But the bill could face parliamentary hurdles in the House, where it is expected to be taken up today in the House Rules Committee.

The full House and Senate could vote on the bill as early as next week. If passed, it would be the first major banking legislation since 1982.

The compromise followed days of meetings between Treasury Secretary James A. Baker III and House and Senate banking committee leaders that lasted until yesterday. In the early evening, Baker sent a letter to Senate Banking Committee Chairman William Proxmire (D-Wis.) outlining the changes to the bill that Congress had agreed to make in exchange for a presidential promise that he will not veto the legislation.

The changes agreed to yesterday would: Provide $10.8 billion in additional funds to the FSLIC. Of that total, $800,000 would be refunded to S&Ls to reimburse them for a secondary reserve that had been wiped out earlier this year when the fund was declared insolvent by federal auditors. The remaining $10 billion would be used to protect insured depositors and close or sell hundreds of hopelessly insolvent S&Ls.

The money is $5 billion less than originally sought by the White House. But it was a blow to the U.S. League of Savings Institutions, the largest lobby group for the industry, which had sought only $5 billion.

The conferees left intact a provision that permits the FSLIC to spend no more than $3.75 billion each year to close ailing S&Ls. Allow nonbank banks and securities firms to buy failed S&Ls with at least $500 million in assets. After March 1, 1988, they could buy failed S&Ls of any size.

Conferees made it clear that S&LS and commercial banks still would have first priority in bidding for failed S&Ls, but that other financial institutions could join the bidding in second place. Nonbank banks are institutions that use a loophole in a federal law that defines a bank as an institution providing checking accounts and commercial loans. By offering one or the other, but not both, of those services, retailers and securities firms have been able to open banks and avoid federal restrictions.

The administration has argued that this provision, considered a major victory for the securities industry, would increase the number of potential buyers for sick S&Ls. Also phase out over several years a mandate that federal regulators go easy on S&Ls that suffering from economic downturns rather than bad management or fraud.

The conferees left in place a provision requiring all S&Ls to adopt generally accepted accounting principles by 1993.

The conferees left untouched portions of the banking bill that would ban nonbank banks created after March 5. Also, they left unchanged a grandfather provision that allows 168 companies that continued to own nonbank banks created before March 5 but would limit the asset growth of those nonbank banks to 7 percent a year after the law takes effect.

The bill would impose a seven-month freeze that would prevent banks from engaging in any securities activities they were not already doing by March 5 of this year. When the freeze expires on March 1, 1988, banks presumably would be allowed to continue their efforts to get regulators to reinterpret existing law so they could expand their products into securities underwriting.