Ivan Selin, one of the Pentagon "whiz kids" who founded American Management Systems Inc. 18 years ago and helped turn it into one of the region's leading computer and systems engineering technology firms, said yesterday that he intends to leave the company within two years.

Selin, 50, who has been chairman of Arlington-based AMS since its creation, said that in two years "I will have been there 20 years. That's four times as long as I've ever stayed anyplace." He said entrepreneurs are always warned about the risks of startups but "no one warns you if your business is a success it could end up a life sentence."

He said AMS was performing so well under the management of President Charles O. Rossotti that he no longer felt challenged. Selin has been involved in a wide variety of activities ranging from studies on drug use in Washington to technology gains in the Soviet Union, and said that he has found his outside activities more interesting than his work at AMS.

"The company is in its best shape ever and the management is very sound. For me personally, after 18 years in the same job, it is time to do something different while I am still relatively young," said Selin, who will remain a member of the AMS board.

Selin said he has not decided what he will do after leaving AMS, but is interested in three general areas. One is his long-standing interest in government and public sector work. When he came to Washington in 1965, Selin was one of the "whiz kids" brought in under Robert S. McNamara to introduce the orderly "systems analysis" approach to the Pentagon. He stayed at the Pentagon until 1970, when at age 32 he left his position as acting assistant secretary of defense for systems analysis to help set up AMS.

Selin said he is also exploring the private sector but he would like to work with "much newer technology" -- such as supercomputers and superconductors -- than he has been involved with at AMS. He said a third area of interest would be some work that would enable him to travel extensively or live overseas. Selin said he speaks eight languages, has lived in Europe and has expertise in the Soviet Union and the Pacific Basin affairs.

He said money would not be a primary consideration. "I have as much money as I need," he said. He declined to say how much he was worth, but did say that he owns 1 million shares of AMS stock, which closed yesterday at $17.37 1/2 a share.

AMS develops, installs and operates computer software systems for the government and business. It also offers management and information technology consulting services.

In recent times, AMS has grown impressively. Net income was $7.6 million fiscal 1987, up 45 percent over the previous year, and revenue was $174 million, up 29 percent.

Rossotti said he does not expect Selin's departure to change the direction of AMS.