Members of the House and Senate banking committees, bowing to pressure from the White House, agreed last night to give $10.8 billion to the nation's insolvent S&L insurance fund and to allow securities firms and "nonbank banks" to buy failed S&Ls.

The lawmakers reached the agreement by reopening a House and Senate conference committee, which had agreed to a compromise banking bill three weeks ago that provided only $8.5 billion to the Federal Savings and Loan Insurance Corp.

The highly unusual parliamentary move of reopening a conference was undertaken to avoid a threatened veto by the White House. The administration has warned that President Reagan would veto the original conference bill because he thought it was anticompetitive and failed to provide enough money for the insurance fund. The fund has been drained by a record number of failed S&L in recent years.

By reopening the conference committee to change the banking bill, the chairmen of the House and Senate banking committees ensured that the full Senate will not be able to amend the bill, according to congressional aides. But the bill could face parliamentary hurdles in the House, where it is expected to be taken up as early as today in the House Rules Committee.

The full House and Senate could vote on the bill as early as next week. If passed, it would be the first major banking legislation since 1982.

The compromise followed days of meetings between Treasury Secretary James A. Baker III and House and Senate banking committee leaders that lasted until late yesterday. In the early evening, Baker sent a letter to Senate Banking Committee Chairman William Proxmire (D-Wis.) outlining the changes to the bill that Congress had agreed to make in exchange for a promise that the president would not veto the legislation.

The changes agreed to yesterday would: Provide $10.8 billion in industry funds to FSLIC. About $800 million would be used to reimburse several dozen S&Ls for money contributed to a special reserve in the federal deposit fund that was wiped out earlier this year when the fund was declared insolvent. The remaining $10 billion would be used to protect insured depositors at hundreds of hopelessly insolvent S&Ls that the government can now not afford to close.

The money is almost $5 billion less than the $15 billion the White House wanted. But it is twice the amount lobbied for by the U.S. League of Savings Institutions, the largest S&L lobby group. Industry insiders say the league wanted the smaller amount so that the problems would be pushed into the next administration, which might be more sympathetic to a taxpayer bailout.

The conferees left intact a provision of the compromise bill that permits FSLIC to spend no more than $3.75 billion each year to close ailing S&Ls. Allow nonbank banks and securities firms to buy failed S&Ls with at least $500 million in assets. After March 1, 1988, they could buy failed S&Ls of any size.

The administration has argued that this provision, considered a major victory for the securities industry, would increase the number of potential buyers for sick S&Ls.

Conferees made it clear that S&Ls and commercial banks still would have first priority in bidding for failed S&Ls, but that other financial institutions could join the bidding in second place. Nonbank banks are institutions that offer checking or commercial loans, but not both, and thus avoid restrictions on who can own a bank and where.

For 50 years Congress has sought to bar securities firms and other nonbanking businesses from owning banks. To sidestep that restriction, retailers and securities firms in recent years have used the nonbank bank loophole to enter consumer banking markets.

Also phase out over several years a mandate that federal regulators go easy on S&Ls suffering from economic downturns rather than bad management or fraud.

The conferees left in place a provision requiring all S&Ls to adopt by 1993 generally accepted accounting principles, which are stricter than the relaxed rules they have been allowed to use since the early 1980s.

The conferees left untouched portions of the banking bill that would ban nonbank banks created after March 5. Also, they left unchanged a grandfather provision that allows 168 companies to continue to own nonbank banks created before March 5 but that would limit the asset growth of those nonbank banks to 7 percent a year after the law takes effect.

The bill would impose a seven-month freeze that would prevent banks from engaging in any securities activities they were not already doing by March 5, 1987. When the freeze expires on March 1, 1988, Congress intends to have passed additional legislation defining how far the banking business should be allowed to mingle with securities and other financial service industries.