Buckley family interests and associates yesterday settled a complaint growing out of a lengthly investigation by the Securities and Exchange Commission that had alleged numerous instances of financial self-dealing.

The defendants, including John W. Buckley, who manages the family's finances, entered into a settlement with the SEC at U.S. District Court here. While neither admitting nor denying the allegations contained in the 39-page civil complaint, the defendants agreed to take remedial actions.

Also named as defendents, Buckley's brother-in-law and business associate, Benjamin W. Heath, and Washington attorney C. Dean Reasoner, a long-time adviser to the Buckleys, were enjoined with Buckley from making untrue statements to public stockholders in violation of the federal securities laws.

Central to the SEC complaint is the Buckley family business, Catawba Corp., which was founded in 1950 by the late William F. Buckley Sr. and was based in New York until it ceased doing business after the SEC probe began. Buckley's eight surviving children and their heirs, including columnist William F. Buckley Jr. and Reagan administration official James Buckley, each had a 9.8 percent interest in Catawba.

Catawba is named as a defendant, as are four companies in which Catawba had an interest. They are Canada Southern Petroleum Ltd., Coastal Caribbean Oils and Mineral Ltd., Magellan Petroleum Corp. and Pantepec International Inc.

Catawba provided various technical services to the defendant companies and other publicly owned firms.

The SEC alleges that between 1969 and 1980, the Buckley interests not only controlled Catawba but also the publicly owned companies as well. According to the SEC, this allowed Catawba to engage in numerous instances of self-dealing to the detriment of the publicly owned companies' stockholders, who were not informed of the transactions.

According to the SEC complaint:

* Catawba collected $6.1 million in fees from the public companies without supplying adequate records of work done. The SEC said that the same officers at Catawba, who billed the public companies, were at the same time in the management of the public companies and "approved the invoices for payment and caused them to be paid."

* Between 1972 and 1978, the public companies paid about $570,000 toward maintenance of the Buckley family estate in Sharon, Conn.

When the Buckley interests in 1975 sold a subsidiary of United Canso Oil & Gas Ltd., with its drilling interest in the North and Celtic seas, Catawba figured it should get its standard royalty payment of one-64th of revenues from drilling, even though no such revenues had been gained at the time of the sale. Stockholders of United Canso were told that the $3.2 million payment to Catawba was recommended by a special committee, but they were not told that the two-man committee was made up of a Buckley brother-in-law and a former banking associate of the family.

On numerous occasions before they sold United Canso, the Buckley interests caused that successful firm to invest in failing businesses also controlled by the Buckleys. The payments then allegedly were used to pay debts accumulated by the failing firm with Catawba and with Reason's Washington law firm to the failing firm.

As part of the settlement terms, Catawba agreed to pay $175,000 to an escrow account for distribution to several companies that did business with the Buckley firm. Catawba also agreed to return $600,000 in royalties to United Canso.