Facet Enterprises, an obscure Oklahoma company whose lawsuit against the Pension Benefit Guarantee Corp. threatened to undermine the nation's entire pension insurance system, settled the case yesterday in an unusual agreement that will give PBGC a share of Facet's future profits.

The PBGC agreed to assume the pension liabilities of some 1,400 retirees who never worked for Facet but whose pensions became Facet's responsibility when the company was spun off from the Bendix Corp. on orders of the Federal Trade Commission.

In exchange, Facet agreed to pay the PBGC $13.3 million over 10 years, give PBGC a share of its profits beginning in 1986 and, most important, drop the lawsuit, which could have forced PBGC to assume billions of dollars in liabilities from other pension plans if Facet had won.

The PBGC was created by Congress in 1974 to provide the same protection for pension plans that the Federal Deposit Insurance Corp. does for bank accounts.

"We have a whole new avenue of earnings potential," Facet President James R. Malone said in a telephone interview from his office in Tulsa. He said he expected Facet stock, which has been depressed for several years because of Wall Street analysts' unhappiness over the pension issue, should rise from its current price of about $5 per share when trading resumes on Monday.

Facet, a six-year-old maker of industrial filters and auto parts, reported net income of $1.685 million on sales of $142.8 million in 1981.

When Facet was set up as the result of an antitrust case against Bendix, it assumed the responsibility for the pension plan not only of its workers but of many who had already retired. Twice Facet extracted additional pension funds from Bendix, but its unfunded liability remained between $17 million and $28 million, depending on interest rate projections.

Unable to attain consistent profitability while supporting pension outlays of some $3.8 million a year, Facet terminated the pension plan, with the consent of the United Auto Workers Union, and set up a new one for active employes. The pension obligation of the retirees would have been shifted to their insurance agency, the PBGC.

Under the 1974 pension law, corporations have the right to terminate pension plans, but the PBGC argued that provision was intended to cover bankrupt or insolvent firms, not profitable corporations like Facet. The former executive director, Robert E. Nagle, refused to approve the termination of the plan, saying that if Facet were allowed to unload its obligations on PBGC, which derives its income from premiums paid by corporations, other companies would do so as well.