Nicholas A. Veliotes, the U.S. ambassador in Cairo who gained international notice last fall by demanding that Egypt "prosecute those sons of bitches" responsible for murdering an American tourist in the Achille Lauro hijacking, is retiring from the Foreign Service, the State Department said yesterday.

Department officials said Veliotes, 57, will be president of the New York-based Association of American Publishers. He became ambassador to Egypt in October 1983 after two years as assistant secretary of state for Mideast affairs.

The officials said discussions about Veliotes' successor as ambassador to America's principal Arab-world ally are under way between Secretary of State George P. Shultz and the White House.

They said Shultz has recommended two career diplomats; although their names were being kept secret, sources identified one as Frank G. Wisner, deputy assistant secretary for African affairs.

The officials noted that White House practice is also to consider candidates from outside the Foreign Service before President Reagan makes a choice.

Veliotes made his angry demand for prosecution of the Achille Lauro hijackers over an open ship-to-shore radio after he boarded the Italian cruise ship last October and learned that the Palestinian hijackers had killed Leon Klinghoffer, a handicapped tourist from New York, and had thrown his body and wheelchair overboard.

In a move that severely strained U.S. relations with Egypt, American jet fighters subsequently forced an Egyptian airliner carrying the terrorists to land in Italy, where they now are being tried for the hijacking. The administration later made a number of conciliatory gestures to placate Egyptian President Hosni Mubarak.

The Associated Press quoted an unidentified non-Mideast diplomat yesterday as saying Veliotes was forced out as part of the effort to smooth relations with Cairo and engaged in a "shouting match" when Shultz wanted him to return here for consultations.

However, that account was disputed by State Department officials and other diplomatic sources. Some of these sources said it was true that Egyptian officials were angered by what they regarded as Veliotes' blunt manner and his actions during the Achille Lauro incident. But the sources said he had been obeying Washington's orders and they were unaware of any dispute with Shultz.

In his new job, Veliotes will succeed Townsend Hoopes, another retired diplomat.