Officials of International Business Machines Corp. feared that the marketing of an IBM telephone-switching device in the United States could result in a substantial loss of business with American Telephone & Telegraph Co., a jury here was told today in an antitrust case brought against AT&T by Litton Industries.

In evidence presented by Litton attorneys, documents also indicated a fear of the IBM marketing scheme among representitives of AT&T's Western Electric Co. subsidiary.

At issue is a charge raised by Litton -- which is suing AT&T for $600 million -- that AT&T applied pressure to IBM that may have played a role in an IBM decision not to market the device, called Caranation, domestically in the early 1970s. Litton is using the charge in its lawsuit, in which the company says it was driven out of the equipment field during the same period as a result of anticompetitive tactics by AT&T.

One IBM document, from IBM official P.W. Terrell on Jan. 4, 1971, warned that AT&T "has had the unofficial desire to reduce IBM's share of AT&T's data-processing business. . . . They are concerned about the amount of business IBM has -- the giant and the giant. The Caranation announcement will be another story.

"In my opinion, the announcement of Carantion would seriously hinder our efforts to market System/7 for plant applications. . . . The impact of Carantion will be heavily negative," he wrote. The documents, introduced while IBM Vice President Dante N. Piccone was a witness, show a deep tension at several Western Electric sites about IBM's plans to market the PBX equipment in the U.S. after testing in Europe. Piccone had been involved in sales to AT&T.

In fact, Terrell told Piccone in a March 7, 1972, memo of a meeting with Don Johnson, an assistant vice president with Michigan Bell Telephone Co., after which Terrell concluded that IBM's "entire DP (data-processing) relationship" with AT&T was having "serious" problems.

"After a friendly discussion of about one and a half hours, he literally leapt across the table to tell me how upset he was about IBM going into the switchboard business," Terrell, then a Piccone subordinate, wrote. "Quite frankly, he got violent, emotional . . . he was really upset."

The timing of the meetings recorded in dozens of documents introduced today is particularly important. IBM decided to drop plans to market Carnation in the U.S. in March 1973, just before AT&T signed a major contract for the computers, known as System/7.

Estimates of that contract by Piccone suggested that it ultimately could be worth $353 million, although the contract, according to IBM, proved to be a $38 million transaction.

Piccone consistently refused to link the two decisions. IBM has asserted that the decision on Carnation was based on the product's difficulties in the European market and that the tension between personnel of the two firms was based on problems AT&T and IBM faced in negotiating a new cross-licensing agreement.

AT&T attorney Leonard Joseph repeatedly sought to bat Litton attorney William Simon from introducing the controversial documents.