Justice Potter Stewart, who has often cast the swing vote in the Supreme Court on controversial social issues, yesterday announced his retirement and gave President Reagan the first high court appointment since 1975.

Stewart, 66, gave no specific reason publicly, except to tell the president in a May 18 letter released yesterday that "it is time to go." His wife, Mary Ann, said that "we wanted to have more time together, to enjoy the children and the grandchildren. It is wonderful to retire when you're in good health." Stewart scheduled a news conference for this morning.

Stewart, appointed in 1958 by President Eisenhower, is the second most senior member of a court that has five men over the age of 70.

He set July 3 as the effective date of retirement, after the court concludes its current term. He may extend the date in order to participate in the Iranian assets case recently accepted for emergency consideration. Stewart will go on "senior status," allowing him to serve occasionally as a lower court judge. But, as he told his colleagues yesterday, "I cannot look forward to serving ever again with you."

The Reagan administration immediately began the search for a new justice. Among the likely candidates are three of the president's best friends, White House counselor Edwin Meese III, Attorney General William French Smith and Deputy Secretary of State William P. Clark, as well as former U.S. solicitor general Robert H. Bork. Reagan also said during his campaign that he would name the first woman Supreme Court justice though not necessarily to the first vacancy.

Replacing Stewart will be one of the most critical decisions of the Reagan presidency. Many of the controversial social questions are now decided by one-vote margins on the court and some by pluralities. Stewart has often been described as a swing vote that can go either way. Recently, however, he has been pleasing conservatives more than liberals on questions involving abortion funding, government affirmative actioin, federal regulation and women's rights.

Stewart's announcement was unexpected. Harvard Law School Professor Laurence H. Tribe said that just a month and a half ago he had chatted with Stewart as he does every year about recommendations for law clerks for the court's next term. Stewart went ahead and hired clerks for the term beginning next October. They were informed yesterday of his decision.

Stewart informed Reagan on May 18, in person and in a letter. "I leave with the hope that the Supreme Court will be in good and wise hands, as it works in the years ahead to preserve and protect the constitutional structure of our Republic," the letter said.

Stewart, second in seniority to Justice William J. Brennan jr., yesterday sent a letter to his colleagues on the court. "This is not the time to try to say what these years as a member of the court have meant to me," it said. "Probably you know. Let me only thank each one of you for your friendship and your help."

Stewart has had a slightly more public profile than many justices. On Jan. 20, he swore in his close friend, George Bush, as vice president, a function ordinarily performed by the chief justice. (Stewart reportedly turned down the chief justice's job following the retirement of Earl Warren.) On the bench, he has been known to sneak off behind the curtains to smoked a cigarette between sharp questions he delivers to lawyers before the court in a crisp and loud voice, leaning forward them in his swivel chair.

Mary Ann Stewart and current clerks said Stewart was in good health. "I modestly hope that a good deal of it [the reason for the retirement] is to be with me and the grandchildren," she said.

"He was terribly young [43] when he went on the court and now he is looking forward to other things.It's a great big world there."

Stewart became eligible last January to retire at full salary; he is independently wealthy and has numerous business investments.