The Reagan administration, in the latest of increasingly hard warnings against military intervention during the unrest in South Korea, yesterday cautioned the nation's military commanders not to mount a coup.

The unusually direct statement issued at the State Department did not reflect any new or specific reports that military commanders are plotting a coup, officials said.

But it reflects a concern that predates the current wave of demonstrations: that strong and well-organized Korean military commanders might be tempted to step in to "restore order" to the Korean political scene and, in the process, seize power as they did in 1961 and 1979-80.

"In our view, military intervention would be a serious disservice to Korea's interests," said State Department spokeswoman Phyllis Oakley.

"Recent signs of flexibility on the part of both the government and the opposition lend hope that resolution can occur via positive political actions, not negative military ones."

In her unsolicited statement, Oakley went on to say, "We urge Korean military commanders to concentrate on the defense of Korea and allow the political process to develop in a manner agreeable to the Korean people."

In another military-related remark, a White House spokesman told reporters traveling with President Reagan to Melbourne, Fla.: "We oppose the use of martial law" in Korea.

There was no word from Washington about the U.S. position regarding the possible domestic use of South Korean military forces committed to defend against North Korea under the U.S.-South Korean joint command.

A senior State Department official said the extent of U.S. control over such troops is a difficult legal point, but declined further explanation.

The issue of U.S. control of or complicity in the use of South Korean forces domestically was a matter of dispute and intense political sensitivity in past episodes.

Both White House and State Department spokesmen expressed hope that the new moves toward political dialogue in South Korea will lead to a peaceful resolution of the issues at the heart of the massive nationwide demonstrations of the past two weeks.

But senior officials were guarded in optimism, saying that the situation remains complicated and far from easily resolved.

President Chun Doo Hwan's reported willingness to meet opposition leader Kim Young Sam, to ease the house arrest of opposition leader Kim Dae Jung and to release political prisoners represents the sort of actions Washington has been requesting with increasing intensity in the past few days.

But Chun's move to the path of compromise and concessions rather than repression could be controversial within the South Korean power structure, especially with some in the military, according to experts here.

Reagan's letter to Chun, which was delivered to the Korean president Friday, is reported by U.S. sources to have contained four main recommendations: a renewal of dialogue with the opposition, continued restraint in handling the street demonstrations, freeing of political prisoners and lifting of restrictions on the opposition.

The last point, which was an indirect way of saying release Kim Dae June from house arrest, has been repeated several times in messages to Seoul, officials said.

In a variety of private and public messages, the administration has been speaking with increasing clarity about the sort of political dialogue for which it is asking.

Immediately after April 13, when Chun called a halt to discussions with the opposition about revising the constitution under which his successor would be chosen, the U.S. administration continued to call for "dialogue" and "compromise."

But it did not ask Chun to reverse his decision. In private, officials said it was unlikely that Chun would back down after having taken such a strong and controversial position.

After the recent demonstrations began, however, Washington began subtly to suggest a resumption of the dialogue on the constitutional process -- in effect calling for Chun to reverse himself.

The call has been more blunt in private messages, according to officials.

Secretary of State George P. Shultz, at a news conference in Australia yesterday, sounded that message.

He said, "What we believe should happen is a resumption of talks not simply about the violence, but about the process through which a transfer of power will take place in a way that reflects the will of the Korean people {in} a democratic manner."