A dispute over frozen potatoes could become a hot potato in Nestle' S.A.'s planned $3 billion takeover of Carnation Co.

Nestle''s acquisition of Carnation's frozen-potato business could run afoul of a 5-year-old agreement with the Federal Trade Commission that limits Nestle''s acquisition of frozen-food makers for antitrust reasons. On the other hand, the terms of the agreement might exempt frozen potatoes, solving that problem, but still leaving Nestle' open to the usual antitrust scrutiny by the FTC or the Justice Department.

Then again, if the whole thing drags on until early January, the agreement with the FTC will expire -- taking the heat off those frozen potatoes.

No matter how you slice it, the frozen-potato problem is creating jitters on Wall Street. Industry analysts downplay the problem, but Carnation stock closed yesterday at $78.625 a share -- more than $4 below Nestle''s $83 offering price, indicating that some Carnation investors think they'll never get Nestle''s money. Usually, a takeover candidate's stock trades at very near the offering price.

The problem lies in a consent order Nestle' signed in 1979, allowing the company to acquire the Stouffer frozen-food business. Among other things, the agreement prohibited Nestle' from making a large acquisition in the frozen-food industry without the FTC's approval. "Large" was defined as a frozen-food operation with sales of more than $10 million a year. The agreement doesn't necessarily mean that the FTC would block a new foray by Nestle' into frozen foods, but it does require the commission to review such a move for possible antitrust violations, and perhaps recommend some divestitures of newly acquired operations.

Carnation is primarily a dairy company. But it also has a subsidiary that makes and sells frozen potatoes for use in restaurants, hospital and other institutional food operations. Carnation won't say, but analysts believe the company's annual revenue from frozen potatoes is slightly in excess of $10 million.

That would trigger the consent order. But the order seems to exempt frozen vegetables. It only applies to frozen entre'es and "frozen prepared foods" -- which it defines as "frozen foods . . . cooked or processed in some manner beyond the blanching of vegetables and fruits in the freezing process."

Industry analysts aren't convinced that wording covers Carnation's frozen potatoes. But even if it does, the agreement is set to expire the first week in January, meaning that if the FTC reviews the acquisition, the question of whether it would cause antitrust problems under the agreement could be moot before the review is completed.

"You're only going to slow it up to January, and then the order runs out," said Smith Barney analyst Ronald Morrow, who added, "I just can't believe the FTC is going to give them a problem on it."

The FTC, for the most part, won't comment on the question. But a commission spokesman pointed out that the Nestle'-Carnation deal will still face the usual antitrust scrutiny, regardless of the outcome of the potato debate.