In the largest dollar settlement yet reached by the Department of Energy in its complex oil industry pricing investigation, Kerr-McGee Corp. yesterday agreed to refund $52.6 million to consumers of gasoline, home heating oil and other petroleum products.

In signing the DOE consent decree, Kerr-McGee becomes the first of 34 oil refiners under investigation since December 1977 to settle completely with the government.

The agreement also marks the first time DOE has set out guidelines under which the company will refund much of the money directly to consumers through future price reductions. Retail prices for the company's gasoline may fall 2 or 3 cents.

The decision will have little direct impact on Washington-area consumers. Kerr-McGee operates through 1,800 retail outlets in 34 states, mostly in the South and Midwest. The company markets gasoline under Kerr-McGee and Deep Rock brand names.

The settlement came in the form of a decree in which the company does not admit any regulatory violations.

The DOE's Office of Special Counsel, which is conducting the continuing 14-month-old audit of all major oil companies, had alleged that Kerr-McGee had overcharged by $80 million its wholesale and retail customers for motor gasoline, home heating oil and other refinery products.

Under the terms of the settlement, Kerr-McGee could actually lose more than $52.6 million. It also will forfeit some $8 million in "banked" credits it could have gained as part of the present government system of price controls. If gasoline prices are decontrolled at the pump soon, however, the company would lose those credits anyway.

The refund program agreed upon will cover purchases of motor gasoline ($31.5 million), home heating oil and distillates ( $8 million), general refinery products ($5.1 million) and jet fuel sold exclusively to the Department of Defense ($1.4 million).

The other $6.6 million will go to selected resellers based on an earlier agreement.

The plan ensures that the greater portion of the refunds will go to final retail consumers through a series of price reductions at Kerr-McGee's retail outlets.

The refunds will be made as direct cash payments to resellers (whole-salers and retailers who purchased the products). Those businesses will be allowed to keep between 30 and 45 percent of the refund, then must reduce future prices for such products as gasoline and home heating oil until consumers have realized savings equal to at least the remaining 55 percent of the alleged overcharges.

DOE special counsel Paul Bloom said the settlement "represents and important innovation in the process of resolving audits of the major refiners." He said his office is mandated to complete the audits of 15 of the 34 major U.S. refiners and their pricing policies since the 1973 advent of pricing regulations, and initiate any appropriate enforcement actions by December 1979. He has already charged several major refiners with overcharges totaling more than $2.2 billion.

Bloom called the Kerr-McGee consent order "significant" because it "relies principally on marketplace, rather than judicial or administrative, arrangements to make restitution to possibly overcharged consumers."

He also said that while participating distributors, jobbers and other middlemen will be compensated for their possible injuries, "they will not enjoy windfalls at the expense of ultimate consumers, who probably bore the greater part of any possible overcharge."

Further, Bloom claimed, this plan places "little burden" on the government and retail consumers to recoup their losses, "as no administrative claims system will have to be established."

The Kerr-McGee order cannot, under DOE regulations, be made final until the public has had 30 days to comment, and the agency evaluates those comments.