Warren E. Burger, the 15th chief justice of the United States, will retire next month after 17 years on the court and 61-year-old Justice William H. Rehnquist will be nominated to replace him, President Reagan announced yesterday.

At an unexpected news conference, Reagan said he would nominate Antonin Scalia, 50, a conservative activist on the U.S. Court of Appeals here, to fill Rehnquist's seat if he is confirmed as chief justice by the Senate.

The change of Scalia for Burger is expected to have little ideological impact on a closely divided court that has for years been dominated by a group of moderate, centrist justices. Burger and Rehnquist have consistently voted together on most issues, although constitutional specialists expect that Scalia, an outspoken conservative on virtually every issue, from abortion to criminal rights to executive branch power, will be even more in tune with Rehnquist's views.

Rehnquist, immensely popular and respected by his colleagues, may prove more intellectually adept than Burger at building and holding conservative majorities and thus sway the court further to the right. In his 17 years as chief justice, Burger had limited success in molding the court to his conservative model.

Burger informed Reagan of his decision to retire on May 27, telling the president -- as he said again yesterday -- that his sole reason for leaving the court was to devote full time to his job as head of the Bicentennial Commission of the United States, which is planning to celebrate the 200th anniversary of the Constitution beginning on Burger's 80th birthday, Sept. 17, 1987. Reagan appointed Burger chairman of the commission a year ago.

Burger, who has been in excellent health recently except for a severe case of the flu this winter, said health played no part in his decision to resign. Asked if he was resigning now so that Reagan would be able to appoint his successor, Burger said that if that had been his goal, he could have waited another year or longer.

The announcement yesterday afternoon came as a shock even to the other justices. The justices had met yesterday morning but neither Burger nor Rehnquist had said anything to their colleagues.

A court messenger delivered copies of the 78-year-old Burger's brief resignation letter approximately 15 minutes before the news conference. The messenger invited the justices, their clerks and staff to the justices' conference room, where two televisions had been set up.

Only when they saw him on television standing at the lecturn of the White House news room did Rehnquist's colleagues realize that he had been selected to be their new chief. Similarly, they had no advance warning that Scalia, an appeals court judge for four years and a law professor and justice department official before that, would be nominated as their new colleague. The crowd of about 50 was "flabbergasted" according to one of those there.

Burger, who had been devoting dozens of hours a week to bicentennial activities, had confided that the work was taking up too much of his time and that he might have to give up one job or the other.

Although his choice struck court observers as unusual, several of Burger's longtime acquaintances said that Burger was a natural politician who has always preferred the political and administrative aspects of his job to the drier, intellectual aspects.

Burger cared deeply about the bicentennial and viewed it as the capstone of a judicial career that began when President Dwight D. Eisenhower appointed him to the federal appeals court here in 1956. Burger was known to be deeply troubled that the bicentennial effort was lagging in terms of fund-raising, visibility and public support.

Burger may have felt that he had accomplished about all he was going to on the court, according to one source who has known him for years. He may have already made a choice to retire in the next year or so, that source said, so a central question for him was whether he would retire to head a successful bicentennial celebration or a failure.

There also has been speculation in the last two years that Burger would retire during Reagan's second term to give the president the choice of a new chief justice.

In an evening news conference, a relaxed Burger spoke casually about his future plans to write, paint and spend more time with his wife. "I'm not going to let you guys write all the books," he told the assembled news media throng. "I've got three or four books I want to write."

"I have a lot of other things I want to do," he said at another point. "I have some pictures to paint." He added, "I never had any ambition to be a judge. I loved practicing law. If tradition didn't prohibit it, I'd love to go back to practicing law."

President Nixon, who had made the Supreme Court's liberal reign under Chief Justice Earl Warren a campaign issue, appointed Burger to replace Warren in 1969.

But under Burger's leadership the court has left the great bulk of the Warren Court legacy in civil rights, federal power over the states and other areas virtually intact. Headed but never really led by Burger, the court continued to expand on many of those rulings, ordering school busing for desegregation and establishing a constitutional right to abortion.

As chief, Burger devoted his energies to modernizing the judiciary and encouraging improvements in a creaky administrative structure. He presided over the growth of a federal judiciary with a handful of employes to one with a budget of over $1 billion.

Burger also continued his many outside activities such as his membership on the Smithsonian board of directors and his leadership with the American Bar Association. His major projects for years were prison reform and court modernization.

Asked yesterday what he regretted not finishing during his tenure, Burger selected an administrative goal, not a legal one. He singled out his inability to convince Congress, after substantial lobbying, to set up an intermediate court between the courts of appeal and the Supreme Court to relieve the high court of some of its workload.

Knowledgeable sources said yesterday that they expect substantial changes in both substance and style under Rehnquist. Rehnquist is not expected to devote nearly as much time to travel, speeches, politics and the ceremonial trappings of office as Burger did.

Rehnquist has privately expressed objections to taking on those extrajudicial duties because he has felt they were irrelevant.

The only formal power the chief justice has over his eight colleagues on the court is the right, when in the majority, to assign the drafting of opinions.

Burger had provoked the lasting resentment of several justices who accused him of trying to manipulate the court by reserving his vote until after the others had indicated their leanings, so that he could say he was in the majority and assign the opinion to the most conservative justice in the majority. He was also known to occasionally harangue his colleagues in criminal cases.

Sources yesterday said it was inconceivable that Rehnquist would pass rather than state his views on a case, even though he, like Burger, will often find himself dissenting from majority opinions.

To fundamentally alter the ideological coloration of the court, Reagan will have to have an opportunity to replace one of the liberals or moderates with another appointee of the Rehnquist-Scalia stripe. The two most liberal justices are William J. Brennan Jr., 80, and Thurgood Marshall, who will be 78 in three weeks. Two other justices regarded as moderates are also in their late 70s -- Lewis F. Powell Jr., 78, and Harry A. Blackmun, 77.

Senate Republicans yesterday hailed Reagan's choices of Rehnquist and Scalia and predicted confirmation for both nominees, while Democrats, acknowledging the president's prerogatives to select whomever he wished, promised a "fair but thorough confirmation process."

The nominations come at a time when Republicans control the Senate but fear they could lose that control in November.

Senate Majority leader Robert J. Dole (R-Kan.) said, "I would guess they would be confirmed by this body without a great deal of delay."

Conservatives yesterday hailed the nominations, predicting the changes will lead the court in a more conservative direction. Bruce Fein, a private legal consultant, said Rehnquist has a "congenial personality," and that he would be more of a consensus builder on the court than Burger had been. "He will be an emollient rather than an inflammatory," Fein said. In close cases, he said, Rehnquist's personal diplomacy may bring the court together.

In that way, Fein said, Rehnquist is very similar to Scalia, whose colleagues, even when strongly disagreeing with him, find him a charming and likeable colleague.

Liberal Harvard Law School professor Laurence H. Tribe said both were "distinguished and bright men" and "men of tremendous affability." But Tribe urged the Senate to focus on their positions on individual issues -- such as separation of church and state -- something that is not easy to focus on given that each nominee's "intellectual power is so great."

Tribe said the Senate also should take into account the timing of Burger's departure, which he said was "obviously timed for the November elections, not the bicentennial." Tribe said Burger could not have expected anyone to believe the resignation was for the bicentennial commission. "If anyone does" believe that explanation, Tribe said, "there is a bridge in Brooklyn that is for sale."