Securities firms and nonbank banks could enter the savings and loan business by acquiring failed S&Ls under a compromise proposal being considered by Congress and the White House.

The proposal, which would amend a banking bill recently agreed to by House and Senate conferees, also would give $10.8 billion to the Federal Savings and Loan Insurance Corp., the insolvent federal fund that insures deposits at S&Ls. The bill now provides $8.5 billion, just over half the $15 billion the White House originally requested.

Treasury Secretary James A. Baker III met yesterday with the chairmen of the House and Senate banking committees to try to agree on changes to the bill, sources on both sides said. The changes are intended as a compromise to head off a threatened veto by President Reagan, who believes the current banking bill is anticompetitive and provides insufficient funding for the beleaguered S&L insurance fund.

Congress wants to avoid a veto because it would prolong publicity about possible fraud and other problems in the S&L industry and about the role key Democrats in Congress may have played in trying to help the industry.

House Banking Chairman Fernand St Germain (D-R.I.) is under investigation by the Justice Department for possible improprieties involving entertainment and other expenses that have been paid by the U.S. League of Savings Institutions, the industry's largest trade group.

House Speaker Jim Wright (D-Tex.) has been embarrassed in recent months by disclosures that he telephoned the chairman of the Federal Home Loan Bank Board about the agency's handling of several failed S&LS in Texas.

The White House wants to avoid a veto because it fears Congress might have enough votes to override it.

Congressional aides cautioned yesterday that the proposed compromise may fall apart. Even if a compromise is worked out, the changes would make the bill more vulnerable to being killed by parliamentary maneuvers in the House, they warn.

The banking bill as it now stands would give FSLIC $8.5 billion, would temporarily prevent banks from selling additional securities products and would close the nonbank bank loophole.

The loophole involves a federal law that defines a bank as an institution providing checking accounts and commercial loans. By offering one or the other, but not both, of those services, retailers and securities firms alike have been able to open banks and avoid federal restrictions.

The bill also would permit 168 companies, including Sears, Roebuck & Co. and Merrill Lynch & Co., to keep the nonbank banks they now own. It is these 168, plus securities firms, that would be allowed to buy failed S&Ls with assets of at least $500 million under the compromise plan being worked on by the Treasury and Congress.