Texas Instruments Inc., one of the leaders in the home computer business, said yesterday it will stop making home computers because of mounting losses. The company said it will shut down its home computer production lines next month and take a $330 million write-off that will give TI a $110.8 million loss for the third quarter.

The announcement--expected in the industry--was the latest sign of big trouble among manufacturers of small personal computers. Three of the five major companies in the field are losing money and a fourth is having problems.

TI's withdrawal leaves Commodore International Ltd., Mattel, Radio Shack and Warner Communications' Atari unit as the remaining major players in the market for inexpensive home computers, which generally cost less than $200 and can play games, perform home and personal management functions, and act as word processors.

Between them, Texas Instruments, Mattel and Atari lost more than $500 million in the first half of the year. Atari, too, has been rumored to be considering pulling out of the home computer field, while Mattel has suffered from poor sales.

The most successful company in the field, Commodore, which controls a whopping 40 percent of the market, has been plagued by defects and supply problems.

Coleco Industries repeatedly has delayed shipment of its new $600 Adam home computer systems, prompting fears it will not be in stores in significant quantities in time for Christmas. Sinclair, the first company to break the $100 price barrier in computers, has seen the price of its original model drop to $39 and has had trouble launching a more sophisticated machine.

TI's withdrawal had been expected in recent weeks by some industry observers. Sales of the company's model 99/4A computer have been slumping, the company lost $119 million in the second quarter, and TI earlier abruptly canceled the introduction of a new model.

The company has been particularly hurt by several rounds of price-cutting in the industry--which TI, ironically, began last year. The cuts saw the price of a 99/4A, which originally sold for $1,000, drop from $199 to $99 earlier this year. TI was believed to be selling the machines for less than what they cost to make, counting on making its profits by selling peripheral equipment and software.

"Retail sales of 99/4A home computer consoles and software in the third quarter were disappointing," the company said in a statement issued late yesterday from its Dallas headquarters. "It became apparent that fourth-quarter demand would not be sufficient to prevent large additional losses . . .

"In order to limit further drain on TI, we have made the decision to withdraw from the consumer home computer business," the company said. "The first nine months of 1983 have been the most difficult period in the history of TI. We believe that with the painful, but correct, decision to withdraw from the consumer home computer business, that period has now ended."

The company said production of the 99/4A would end next month, causing "significant personnel reductions." Prices of its remaining Model 99/4A computers will be slashed to clear inventory, the company said.

Home computers account for less than 10 percent of TI's sales, which were $1.01 billion in the third quarter. TI said that it will continue in the higher-priced personal computer market, catering to business and commercial users.