Reagan administration officials said yesterday that quiet diplomacy by the United States helped defuse an explosive situation in South Korea last weekend and that new private and public messages to Seoul are likely because of continuing concern about the political instability there.

Sources said U.S. Ambassador to Korea James Lilley delivered a strong message to the South Korean government through Foreign Minister Choi Kwang Soo last Saturday urging that riot police not storm the Roman Catholic cathedral where students had been holding a highly publicized sit-in most of last week. Storming the cathedral, an action which was believed to have been under consideration, was seen in Washington as potentially explosive.

Sunday night the government of President Chun Doo Hwan withdrew riot police from the area around the cathedral in the heart of Seoul and offered safe passage to student protesters. The students left the cathedral Monday afternoon, defusing the standoff. However, protests against Chun's rule have continued in the capital and provincial cities.

Sources said the White House is considering whether President Reagan should send a private letter to Chun. Officials said the administration has decided for now not to send a special emissary to Seoul. They said they believe such a move would unduly raise public expectations and that quiet diplomacy is likely to be most effective.

At the same time, State Department officials said the administration is moving toward public endorsement of the spirit, if not all the wording, of a congressional resolution introduced yesterday calling for renewed political dialogue in Seoul and renunciation of politically motivated violence by all sides.

Rep. Stephen J. Solarz (D-N.Y.), one of the authors of the resolution, said he would ask for a formal administration position on Thursday when the House Foreign Affairs subcommittee on Asian and Pacific affairs votes on it. Solarz chairs the subcommittee.

U.S. policy at present, which was restated yesterday by State Department spokeswoman Phyllis Oakley, is to call for "progress toward a more open political system" in South Korea. Urging solutions that enjoy "the broad support of the Korean people," Oakley said that "real progress can come only through dialogue and a willingness of all sides to compromise."

A number of officials described the situation in Korea as extremely delicate since Chun decided two months ago to break off dialogue with the political opposition on the constitutional provisions under which his successor is to be chosen and since his selection two weeks ago of former general Roh Tae Woo, his friend and military academy classmate, as presidential candidate of the ruling party and thus his designated successor.

Administration officials were unhappy with Chun's April 13 decision to terminate the discussions on constitutional change but decided there was little they could do to reverse it. Some officials are concerned that Chun might try to use the predictable turbulence that followed to stay in power rather than to step down as promised next February. Others think recent events make this less likely.

Another concern that persists here is that increasing disorders could bring a new military coup in Seoul, eliminating even the facade of civil government that now exists. Such possibilities make caution essential, in the view of U.S. officials.Staff writer John Burgess contributed to this report from Seoul.