The Department of Energy has begun quietly investigating five American companies who this spring dominated the market in gasoline sold outside the normal government-controlled supply channels.

A confidential Energy Department survey of the buying habits of the major oil companies showed that they turned to the five gasoline trading companies for hundreds of millions of gallons a month at a time when the big companies were scrambling to meet supply quotas to their customers.

The survey showed that at a given time the so-called "spot market" purchases totaled "considerably more than 10 percent" of the gasoline in the inventory of some major refineries, according to a source familiar with the survey.

The investigation marks the first comprehensive attempt in the current gasoline crunch to determine if spot market traders improperly benefited from skyrocketing gasoline prices.

The five companies identified in the confidential survey are: Apex Oil Co., based in St. Louis with storage facilities in Baltimore; M&A Petroleum of Victoria, Tex.; Lajet Inc. of Abilene, Tex.; Engelhard Oil Corp., a subsidiary of Philipp Brothers of New York, and Kent Oil and Trading Co. of Houston.

Spokesman for Engelhard and Lajet could not be reached for comment, but spokesmen for the other three companies said they have adhered to federal pricing and allocation rules.

Energy Department investigators are seeking to understand how five small companies could play such a key supply role during the nationwide gasoline shortage. Moreover, the investigators hope to better understand how the spot market functions and whether any of the five companies who dominated the market inflated prices.

Officials for two of the five companies, Apex and Kent Oil, said that they have been visited in recent weeks by Energy Department investigators who were seeking information and records regarding the companies' gasoline sales.

Two of the most puzzling mysteries of the gasoline shortage have been:

How is it that millions of gallons of gasoline are available on the spot market when the government has laid down hundreds of pages of regulations to oversee a mandatory system of allocation between suppliers and customers?

How is it that under a system of price controls, gasoline on the spot market has been going for $1.30 a gallon and more?

The spot market has been variously described as a "gray market," a "surplus market," "second market" or an "auction market."

It represents a significant volume of gasoline each day that can be legally diverted from the normal supply chain under federal rules and is traded by middlemen over a telephone network where a broker's word is his contract.

But it can also represent gasoline that is illegally diverted, federal energy officials say.

The government control over the allocation of gasoline dates to the 1973-74 Arab oil embargo when regulations were written requiring a company that supplied gasoline to a customer during 1972 - called the base year - to continue supplying that customer.

Those regulations were written early this year to substitute 1978 - a year of plentiful gasoline supplies - as the base year.

Because gasoline was plentiful in 1978, and many companies shopped around to purchase their supplies from a variety of sources, the new Energy Department allocation rules helped create a chaotic pattern of supplier-customer relationships.

From this maze of relationships, here are some of the ways that gasoline found its way onto the spot market, according to federal energy officials:

The five middleman companies identified in the Energy Department survey are called resellers and under the department's regulations, they are allowed to keep controll of gasoline designated for customers who have gone out of business or curtailed their usuage. This gasoline often goes into the spot market, officials say.

The reselling companies can also demand from the major refiners additional gasoline allocations for projected growth. The middleman, under the federal rules, can increase their supply by simply stating that their needs have grown and without requirement of proof. Federal officials believe these projects are often exaggerated and help increase the availability of spot-market gasoline.

Federal rules set up a system where the reselling companies can also demand unlimited supplies of gasoline for so-called "priority" customers - farms, dairies and the like - with no requirements for proof of need. Federal officials suspect there have been many abuses under this system and large quantities of gasoline diverted.

A reseller can declare a surplus by telling the government he has offered his gasoline supplies to his customers, but some is left over. The amount left over can thus be sold on the spot market.

The most blantantly improper method of putting gasoline on the spot market is for a selling company to ignore the customers he supplied during the base year and sell all of his gasoline to the highest bidder.

Under this system of mandatory allocation to resellers, major refiners may end up buying back gasoline they were required to supply to the middleman companies, energy officials say.

One senior energy analyst described such aberrations as legal "daisy chains," a term that usually describes the illegal practice of selling oil products on paper through a series of companies while the product never moves - and thus raising the price at each link in the chain.

One major oil company executive said last week that his company was forced to sell gasoline last spring to a middleman company at a price around 65 cents a gallon. "Then they turned around and offered it back to us at 80 cents," the official said. The official said his company refused the offer.

"There's no way that's legal," said one knowledgeable energy, official. Even if the reseller was passing on his own allowable costs and profit margin, the official said the 15 cent mark up was excessive.

One of the five resellers identified in the Energy Department survey was Kent Oil and Trading Co. of Houston.

Kent, which is not incorporated under Texas laws, sells nearly two million gallons of gasoline a day to companies like Exxon, Chevron, Union, Texaco and Gulf, according to Steven R. Kent, the company's president.

Kent said his profits are about a half-cent a gallon. "It doesn't allow for a big bonanza," he said.

Nevertheless, with 15 to 20 employes, Kent said his company's annual sales are between $250 million and $300 million.

One of the companies named in the survey, M&A Petroleum, took strong exception to being characterized as a spot-market seller. "I would say that is incorrect," said Tom McDade, a lawyer with the Houston firm of Fulbright and Jaworski.

McDade represents M&A as well as its founder and driving force, millionaire oilman Albert Alkek. M&A pleaded no contest and paid $100,000 in fines last year for oil-pricing violations in its transactions with Conoco, which was also fined.

In March, Alkek pleaded guilty to concealing pricing fraud in another oil company's operations.

Another of the companies, Lajet, is under investigation by the Department of Justice in a case involving alleged "daisy chain" sales and kickbacks to one of its officers in 1976. Lajet officials could not be reached for comment.

Kent Oil is under investigation by the Energy Department's criminal enforcement office for alleged pricing violations as well as its relationship to Charter Oil Co., whose financing put Kent into business during the 1973 oil crisis. Kent denies any wrongdoing.

Energy Department investigators have been plagued by the growth in the number of companies that have gone into the resell business since 1973, when only 50 companies or less acted as middlemen. Today, there are hundreds of such companies.