The Energy Department is faced with a perplexing problem: What to do with the money it wins back from oil companies accused of overcharging customers.

This is no small problem. Over the past two years DOE has alleged more than $7 billion in such overcharges by the top 15 oil companies stemming from reported violations of comlex federal price and allocation regulations from 1974 to 1976.

And, sources at DOE said yesterday, the department plans this week to accuse four of those companies with an additional billion dollars in over-charges, and expects to end up raising that total to more than $10 billion by the end of the year.

But the most significant number is actually around $660 million, which is the amount that various companies have consented to repay in response to DOE's charges. Most of the allegations have not been resolved, and are the subject of continuing legal and administrative tugs of war.

But in the cases that have been settled, DOE lawyers have attempted to find a way to channel whatever money it collects back to those who deserve it.

In many cases, such an approach is virtually impossible. How, for example, could the department find out the names of the millions of motorists who paid a few cents extra on gasoline at the pump because some refiner reportedly overcharged a wholesaler five months earlier?

But there have been 5 cases, a few involving smaller oil companies, where there have been innovative attempts to help those who had been harmed.

The first settlement leading to a new rebate approach was a case involving Gulf Oil. In July, 1978, Gulf agreed to pay $42.2 million as part of a consent order growing out of charges that the company overcharged crude oil customers.

But instead of paying that money directly into the U.S. Treasury's general fund, the money was placed in an escrow account. And DOE set up an elaborate administrative claims process under which wholesalers or others who can show that they purchased the crude in question could secure refunds.

The second action involved a Feb. 1979 agreement by Kerr-McGee to pay back $52.6 million. In that case, the money was divided three ways. First, cast payments were made to some people who were no longer Kerr-McGee customers. Second, Kerr-McGee actually rolled back prices at its gas stations.Third, DOE ordered refunds through the entire marketing system.

That meant that refunds would go to a wholesaler, provided he agreed to pass on 80 percent of that refund to his customers, who, in turn, had to agree to pass on a smaller percentage to their customers, and so on.

That way, DOE officials pointed out, everyone got something back in a case where it was nearly impossible to figure out who had made money on the original overcharges.

The third case was an Oct. 31, 1979 settlement for $220 million with Cities Service Co.

Because of the enormity of that case, however, the DOE lawyers and the company agreed that a complex administrative rebate system would be too difficult to pull off. Instead, the company agreed to roll back heating oil prices to its customers three cents a gallon from its price on the day the agreement was reached. The rollback will last the length of this winter.

And, the company agreed to forego some future price increases it would have been allowed for gasoline under existing regulations.

The fourth settlement involved a $200 million agreement with Phillips Oil Co. on Nov. 9, 1979. There, the company first agreed to spend $100 million on domestic exploration and development.

Another $75 million came out of what would have been future price increases for gasoline and propane allowed under the regulations.

After $3 million went to direct payments to certain large customers who could show that they were overcharged, the remaining $22 million proved to be the most interesting aspect of the rebate.

Phillips agreed to buy $22 million worth of crude oil on the open market, and use that crude and some of its existing crude to produce more heating oil for its customers. Those customers had faced severe shortage problems, and under this system would be given relief at essentially frozen prices.

The final example of an innovative agreement came only last week, when Getty agreed to pay $25 million into an escrow account that will be used to aid needy people hit particularly hard by rising heating oil prices.

In all cases, DOE spokesmen say, the public reserves its right to sue to recover alleged overcharges, if they feel the DOE arrangement is not equitable.

DOE officials are hoping that many future settlements will involve similar arrangements.

But in the rash of actions expected to be filed by the end of the year, DOE staffers have been unable to take the time necessary to formulate any more such elaborate rebate programs.