Lined on a wall behind an Apple II microcomputer at The Math Box store in Rockville are rows and rows of computer games. Emblazoned with bizarre cartoon characters usually engaged in a struggle to escape something or shoot down intruding projectiles, games such as Ruski Duck sell for between $20 and $50.

Perched in a bookcase behind the protection of glass in the Sears Business Store at the Tycon Business Park in McLean, Va., are five shelves of gray cardboard, booksized boxes with titles such as "accounts receivable," "general ledger" and "inventory control." Inside these packages are avenues to power and success in the form of a floppy disk priced between $300 and $600.

On a video screen attached to an IBM personal computer is the simple equation 1 + 1 = . The space is left for the end-user (computerese for operator) to fill. The self-paced program starts easy and progresses as the end-user learns the rules of addition. The cost, $60.

Entertainment, business and education are the three main catagories of the emerging computer software industry.

Microcomputers have hit the market with stunning success and now software manufactureres are following with packages to keep the machines whirling and consumers buying.

Software companies are springing up and disappearing at a rapid rate in the fight for a niche in a market just beginning to define itself.

Expecting microcomputer software products to follow if not exceed the highly successful pattern set by the home video market, Schwartz Brothers Inc. of Lanham, Md., has created a computer software division along the lines of its successful two-year-old video division.

Schwartz Brothers, owners of Harmony Hut record and tape stores, saw its wholesale video distributorship grow to more than 800 stores in the mid-Atlantic region in two years. Schwartz Brothers was founded in 1946 as a record distributor, opened its first retail outlet in 1969 and now has 25 stores.

Eugene Horn, general manager of the software division, said the computer software market is emerging along the same lines as the video market, with many of the distributors operating "out of basement boiler rooms with '800' toll-free numbers" attempting to meet a huge demand.

"We are looking to be a strong service-oriented distribution center in the mid-Atlantic region," said Horn. "We are going to focus on selling accessories and software products."

Horn said Schwartz Brothers will try to bridge a gap between manufacturer and retailer by screening the currently tumultuous market for products, providing a readily available inventory and establishing a personal sales relationship with dealers.

"We firmly believe the market is unlimited at this time," said Jim Schwartz, president of Schwartz Brothers.

The software market is indeed stampeding across virgin turf. With sales of microcomputers expected to hit more than 1.05 million this year and 2.05 million in 1983, the demand for software will follow and actually exceed hardware growth rates, according to Future Computing Inc., a software marketing research firm in Richardson, Tex.

Future Computing predicts that total personal computer software sales will skyrocket from $500 million in 1981 to $6 billion in 1986 for a compound annual growth rate of 65 percent.

Personal computers are currently divided into three market segments, the $1,000, $3,000 and $10,000 catagories. Future Computing predicts the following software sales for each catagory:

* Software for personal computers under $1,000 is expected to grow from $60 million in sales in 1981 to $2.5 billion in 1986 for a compound annual growth rate of 110 percent. Entertainment software is the largest application for this class of home computer.

* The software market for the $3,000 personal computer should grow from $330 million in 1981 to $2.6 billion in 1986, a 51 percent compound annual growth rate. In this class, productivity software such as word processing, data base and financial planning is the largest application segment. Entertainment software and business software such as accounting are next in size.

The software for the $10,000 catagory will increase from $100 million in 1981 to $1 billion in 1986 for a 55 percent compound annual growth rate. Productivity and business software are the largest application segments.

The expected leaders in software sales during this initial growth period through 1985 are Microsoft, VisiCorp, IBM, Apple Computer and Atari, according to a survey conducted by Future Computing

There are more than 150 computer retail stores in the Washington-Baltimore area with more stores opening every week. One of the largest retail outlets of software is the The Math Box, with six stores. Opened in 1974 as a retailer of calculators, video games and chess games, the Math Box is now shifting almost entirely to the micro computer hardware and software market.

Math Box President Arves Parnes said, ''The president of Apple computers said the biggest obstacle to computers is the guy who uses the pencil and calculator,'' in explaining the company's shift away from calculators and their inherent competition with computers and software.

Parnes is confident the computer is here to stay, predicting that, ''In the future you will have to have a computer in order to be a part of society.''