Genex Corp. of Gaithersburg, the financially troubled biotechnology company, yesterday swallowed a bitter pill in settling its $40 million suit against NutraSweet Co. for a "nominal amount" of cash.

Under the agreement, NutraSweet will drop a claim that Genex overcharged it $1.35 million for amino acids. That amount, previously reported by Genex as a liability, will now be counted as income, according to Genex officials.

NutraSweet will also pay Genex probably between $100,000 and $500,000, according to NutraSweet officials. Genex also dropped its suit against NutraSweet.

After the Skokie, Ill., company failed to renew a contract for Genex to supply L-phenylalanine, one of the two primary ingredients in aspartame, Genex filed suit in October 1985 charging NutraSweet with fraud and violation of federal antitrust, racketeering and securities laws. Genex had invested $17.5 million in a short-lived plant in Paducah, Ky., to increase its productive capacity, and contended NutraSweet had led the company to believe it wanted a long-term supply relationship.

NutraSweet Vice President Thym S. Smith said G.D. Searle & Co. -- then the parent of NutraSweet -- had always intended to cut its dependence on outside supplies of L-phenylalanine. Sales of the amino acid accounted for almost 90 percent of Genex revenue in the first half of 1985.

NutraSweet, now a wholly owned subsidiary of Monsanto, which purchased Searle in 1985, is the sole U.S. maker of aspartame, with 1986 revenue of $711 million. Marketed under the trade names NutraSweet and Equal, aspartame has been widely used as a replacement for sugar and saccharine in soft-drinks and a range of low-calorie foods since it was introduced in 1981.

Gary E. Frashier, who stepped in as Genex president April 1, said the settlement was necessary and that Genex needed to put the problem behind it. Genex is developing a protein-adhesive that could be used in dental surgery, and another genetically engineered protein to be used in medical diagnostics.

Genex stock has been selling far below its historic high of $22.50 since the loss of its primary contract. The stock closed yesterday at $1.25, up $.125 on volume of 18,900 shares