The nation's retailers, who already are suffering from disappointing earnings and high inventories, were warned today that even harder times may lie ahead.

A panel of experts at the National Retail Merchants Association annual convention here told industry executives that consumers will be spending a smaller share of their income on retail goods and services over the next five years.

As a result, general retailers can expect little, if any, growth, and a bruising battle for market share with their competitors, they said.

In addition, the panel said some retailers are certain to go out of business. The most likely targets will be the variety five-and-ten stores. Also projected to do poorly are Sears Roebuck & Co., J. C. Penney and Montgomery Ward Co., whose house-brand apparel will not compete as effectively as the brand-name clothes that will be offered by department stores.

On the other hand, mail-order firms and specialty retailers -- particularly those selling electronic goods or toys -- are likely to fare better than the rest of the industry.

"This suggests all sorts of new attitudes" for retailers, warned William Ress, chairman of the retailing consulting firm C. W. Ress & Associates. Ress; Eleanor May, a professor of business administration at the University of Virginia; and Walter J. Salmon, a Harvard Business School professor, are the authors of a two-year study, "A Look at Retailing to 1990," sponsored by the nonprofit Marketing Science Institute.

According to the study, "growth in retail sales in real dollars will be unexciting in the [rest of the] 1980s," May said. While she said she expects the gross national product to grow at an average annual rate of 3.1 percent, retail sales will grow at a yearly average rate of only 2.3 percent.

Her reason: Americans will be paying more federal and state taxes as well as higher medical and utility bills. Additionally, relatively high interest rates will continue to encourage consumers to save money.

As a result, the percentage of disposable income devoted to sales of merchandise and related services will drop from 63 percent today to 59 percent in 1990 -- a significant decline considering that each percentage-point drop costs retailers $24 billion, May said.

Among the biggest losers will be the apparel market, the panel said. That market alone accounts for about half of the department store business. "Apparel will be a tough game. It's a zero-growth game," said Ress, who attributed the decline to the decreasing number of teenagers.

That age group not only spends a lot of money on clothes, but also plays an important role in setting styles for other age groups. Additionally, Ress said there will be more senior citizens, who traditionally spend less on clothing.

An increase in the number of working women, combined with a rise in the number of children in the population, will help offset these factors, but not enough to turn the Toy sales are expected to climb by more than 7 percent a year. apparel business into a moneymaker over the next five years, Ress said.

Despite the gloomy assessment, the panel predicted there will be some clear winners in the years ahead. Chief among these are electronic sales, which are expected to grow by an average of 14 percent a year. "There has been a technological breakthrough and an acceptance of technology by people," which makes electronics a big winner, Ress said.

Additionally, toy sales are expected to climb by more than 7 percent a year, largely because children of the Baby Boomers are now bearing children of their own. Sales of optical goods also will also be strong because those Americans born in the post-World War II baby boom "have now reached the age where they need glasses to read," Ress said.

The mail-order business also will prosper, Salmon noted, largely because it fulfills a major need of consumers: reduced transaction time at the store. Discount stores also will do well. But because it takes longer to complete a sale at these stores, Salmon does not expect them to do as well as many in the industry have predicted.

He predicted that growth in sales volume at conventional stores will average 2.5 percent over the next five years and that discount stores will see an average increase of only 3.5 percent a year. Catalogue showrooms, on the other hand, will have an even larger increase, he said.