Supreme Court Justice Potter Stewart said yesterday that it was a letter last year from a high school student, asking why he stayed on the court so long, that started him thinking about retiring.

That, and a desire to spend more time with his family, were responsible for his decision to retire after 23 years on the high court, he said. He waited until after the presidential election because he believed the question of his replacement would become a campaign issue. "I thought that would be very harmful to the court and to the country," Stewart told a news conference.

After consulting his wife and his close friend, Vice President Bush, and informing President Reagan, Stewart, 66, made the unexpected announcement Thursday that he would retire effective July 3, giving the president the first Supreme Court appointment since 1975.

In a discussion with reporters yesterday, only the second news conference eve held by a sitting Supreme Court justice (the first was by William O. Douglas after a lengthy illness), Stewart came out against proposed laws to strip the courts of jurisdiction over abortion, school prayer and busing. He also had some advice about the appointment Reagan must now make.

"It's an insult to the court, to the appointee and to the American public to appoint somebody just because he or she is not a white male," Stewart said when asked if Reagan should appoint a woman, as the president has suggested he might. "There is nothing more antithetical" to the process of finding a good judge than " to think it has something to do with representative democracy," Stewart said.

As for ideology, another factor under discussion now, Stewart had this to say: "It is the first duty of a justice to remove his own moral, philosophical, political and religious beliefs and not to think of himself as some great philosopher king and apply his own ideology."

Stewart said that after his dissent in the ruling banning state-sponsored school prayer he was embarrassed, because among the 2,000 letters he received were many attributing his vote to his religious beliefs (he is an Episcopalian). "My dissent was based on my understanding of the U.S. Constitution," he said. A justice's "only boss is the Constitution and the law."

Asked how he would like to be remembered, Stewart first said he couldn't answer that question. Pressed on it, he said: "As a good lawyer who did his best."

The selection process for his replacement has been under way in the administration since March, when Stewart informally told Attorney General William French Smith of his plans and the Justice Department began compiling names of potential candidates. But because of Stewart's desire for secrecy, little advice has been so far sought outside the White House and Justice Department.

(Smith, thought to be one of many likely candidates, came close yesterday to taking himself out of the running. Tom DeCair, Smith's spokesman, said there was no job Smith wanted less than the Surpreme Court slot.)

Donna Gallus, then a high school student in St. Cloud, Minn., wrote Stewart on Feb. 19, 1980, shortly after his 65th birthday made him eligible for retirement at full pay.

Gallus wrote that her class was studying the court. "Dear Mr. Steward [sic]," she said, "my reason for writing to you is I would like to know why you have stayed on the Supreme Court so long. We have learned you have the opportunity to retire, but still you are a judge on the court. I am not saying you need to retire but am asking why you stay on the court longer than you need to."

Stewart wrote back that he had not become eligible for retirement until the previous month. "Under these circumstances, it has not occurred to me, at least so far, that I have remained as a member of the Surpreme Court an unduly long time since the date of my eligibility to retire from it."

Stewart said yesterday that the letter "started me thinking. But I didn't want to do it in an election year and I didn't want to do it in the middle of a term" of court.

"I was the youngest federal judge in the country" when appointed to the Sixth Circuit Court of Appeals in 1954, Stewart said. "I thought it might be a good idea to retire before I became the oldest. . . . I've always been a firm believer in the principle that it's better to go too soon than to stay too long."

Reflecting views expressed in dissents involving affirmative action by government, Stewart strongly objected to appointing a woman because she is a woman. "As far as i'm concerned, the mark of a good justice or a good judge is an opinion you can read and have no idea whether it was written by a man or a woman, a Republican or a Democrat or a Christian or a Jew. . . . The most important thing is quality and competence and temperament and character and diligence," he said.

He said the proposals in Congress to strip the courts of jurisdiction over controversial issues such as abortion, school prayer and busing "concern me. They would immediately present difficult constitutional questions. I'm glad I'm not going to be here to answer them, and I hope this court will never have to answer them.

"I hope no such bill will be enacted."

After the formal news conference, Stewart commented on Brown vs. Board of Education, the 1954 ruling that desegregated schools and led to busing. "That case simply held that it was unconstitutional to keep children out of a public tax-supported school based upon the color of the skin. cIt didn't say anything about there having to be in every school building a proportion of white and non-white students and so on.

"We've come a long way from that decision, and while it was very slow in being implemented in certain areas of the country, I think the basics of it are now accepted and are implemented."

On the concurrence he wrote in the 1963 decision legalizing abortion: "Were the case here again today, I would write it again."

Stewart was also asked about reported disagreements with Chief Justice Warren E. Burger and about reports of personal dissension in the Burger court. "Every person with whom I've served on the court," he said, "I've liked very much. You don't choose them . . . so some you perhaps like better than others. But it's your duty to work with each of them."

His one regret, Stewart said, was the words he used in a famous obscenity case on the subject of defining hard-core pornography. "I know it when I see it," Stewart wrote in the 1964 opinion, "and the picture involved in this case is not that."

"That's going to be on my tombstone," he said yesterday, "out of all the other solid words I've written."