Over-the-counter securities have become the best-performing sector of the stock market to date in 1978.

The 2,550 companies whose stocks are not listed on major exchanges "are performing, and their stocks are beginning to reflect it, said one broker, Charles Detzelberger at Stern Brothers & Co. of Kansas City. "These companies are showing outstanding earnings performances - maybe because the medium-sized companies are not as much plagued by government regulation as the bigger ones are."

"Institutional people who a couple of years ago were totally turned off by medium-sized and smaller companies are coming back to them," added Kenneth Wessels of Piper, Jaffrey & Hopwood in Minneapolis, seeking to explain an explosion of over-the-counter prices and volume since the new year began.

The closely watched Dow Jones average of 30 industrial blue chips on the New York Stock Exchange has seesawed, but its gain since Jan. 1 has been under 2 percent.

The Dow closed down 10.31 to 844.25 compared with 831.17 at the beginning of the year. Some technical analysts of Wall Street prices had forecast such a decline at about this time.

In contrast with the Dow, however, the composite over-the-counter index of the National Association of Securities Dealers Automated Quotations system (NASDAQ) has been moving steadily higher.

The NASDAQ index stood at 124.19 last night, off 0.32 for the day but up some 19 percent from 105.05 last Dec. 30.

In terms of volume of shares traded, the OTC market also has grown more rapidly than the Big Board. Through Wednesday night, volume in 1978 was up 43 percent of OTC issues compared with a 32 percent gain at the New York Stock Exchange.

During the first five months of 1978, Big Board volume was 2.77 billion shares compared with 2.16 billion from 769 million shares.

Volume on the American Exchange rose 28 percent through Wednesday night to a level about one-third that of OTC trading. But the Amex market value index has gained nearly as much as the NASDAQ, up some 18 percent from 127.89 last Dec. 30 to 150.74 last night.

Brokers surveyed around the country offered various explanations for the surge of interest in OTC stocks, including issues of many companies based in the Washington area.

"Some NASDAQ/OTC stocks are selling at one-third replacement value . . . the investor with a keen eye is spotting them," said Frederick Berliner, research analyst at Herzog. Heine & Geduld in Jersey City.

At an E.F. Hutton office in Los Angeles. Richard Goodfellow reported that OTC retail sales are up more than 40 percent. "The old holders are buying more, and people who were sitting on the sidelines are coming in now . . . Four years ago, the institutions as a group decided to get into the big stocks. Since that time, they've found themselves taking some tremendous baths. That smart institutions have found that they can't all do the same things . . ."

One local measure of interest in so-called secondary stocks - those not listed on the NYSE - is Johnston, Lemon & Co.'s index of 30 area companies (which includes some NYSE, Amex and OTC stocks). The Johnson, Lemon indix has been flying high all year and reached a new high of 98.256 on Wednesday before easing 0.33 yesterday to 97.926.

For the year, the local stocks index has increased by almost 22 percent from 80.372 at the end of 1977.

One local OTC issue that has soared is Quality Inns International, up 100 percent since Jan. 1 to $5.25 yesterday in the wake of a forecast turnaround from previous losses.The forecast turned out to be true yesterday, as the Silver Spring firm posted earnings of $2.5 million in the third fiscal quarter compared with a loss of $70,000.

Only two stocks of the local index have declined in 1978 - Washington Real Estate Investment Trust (down 1.6 percent to $23.75, listed on the Amex) and Potomac Electric Power Co. (down 7 percent to $14.625, listed on NYSE). Both stocks are sensitive to higher interest rates.