A two-star Navy admiral representing the Joint Chiefs of Staff on the U.S. strategic arms negotiating team, who was a target of unfavorable comments in a controversial government memo, is retiring from the Navy and will not return to the negotiations with the Soviet Union in Geneva.

Rear Adm. William A. Williams III is the second high-ranking member of the U.S. team at the Strategic Arms Reduction Talks (START) who will not return when the negotiations resume June 8. Jack W. Mendelsohn, who represented the U.S. Arms Control and Disarmament Agency (ACDA) on the delegation and was commented upon unfavorably in the same memo, already had resigned.

It is not clear whether their actions are directly linked to the memo, which was written earlier this year, reportedly by aides to the chief U.S. START negotiator, Edward L. Rowny. But Mendelsohn suggested it had had an indirect effect on his action, and other sources said they believed it also might have played a part in Williams' decision.

The memo was described as "talking points" prepared for Rowny and to be turned over by him to President's Reagan's new choice to head the ACDA, Kenneth L. Adelman. Although Rowny has denied publicly that the memo reflected his views, Adelman eventually passed them along privately to a close associate with the notation that these were "Rowny's very confidential real views on people."

The memo, which has harmed morale on the START negotiating team, according to several State Department officials, depicted three of Rowny's deputies as wanting "progress at any cost" in the talks. Another was described as "solid, but nit-picker" and another as "OTL, abrasive; doesn't want to learn."

Williams was named in press accounts as the aide described as "OTL," meaning "out to lunch."

Navy sources said Williams was "plucked" in February in the annual weeding out of flag officer ranks, which normally leads to retirement.

Williams said only that "I would prefer not to discuss this at this time" and that he had "no clarification" to offer at this time.

Mendelsohn, who was among those characterized in the memo by the phrase "progress at any cost," had told associates before the memo surfaced of his intention to leave the delegation for another post in the State Department. He also declined to comment in detail, but did say that, while he was planning to go, the memo "made it clear I could not stay."

Mendelsohn added that "Those comments about Admiral Williams were totally unjustified."

The memo had added to controversy over Adelman, who was sworn in as ACDA director at a White House ceremony yesterday. Reagan called Adelman, 36, "a man of vision and honor."

Adelman responded that nuclear arms control negotiations would take "time, energy, tenacity and imagination . . . . Your administration has surely chosen the difficult, bold road on arms control, a road which can lead to agreements which make the world a safer place for all of us, a world with fewer weapons on both sides, a world wherein freedom can blossom and diversity can be treasured."