THE UNITED STATES, under the Reagan administration, is determined to stand firm on its rights in dealing with the world's troublemakers. But the case of the North Korean missile is not quite so unambiguous as the attack by the Libyan fighter planes. Incidents like these are, unfortunately, not uncommon along certain boundaries. It is better tactics not to let the American responses set a pattern of immediate and reflexive threats.

The North Korean missile exploded a few miles from the American reconnaissance plane as it flew along the Korean coast. The United States immediately declared that the plane was in international air space and denounced the firing of the missile as illegal. "The United States as a leader of the free world has an obligation to be a strong advocate of adherence to accepted rules of international law and behavior," Secretary of State Haig said at his news conference. Earlier Edwin Meese, the counselor to the president, said that in the event of a repetition, American planes might attack the site from which the missile was launched. Language like that comes pretty close to a dare from which it might be difficult to retreat. Some time ago Mr. Reagan said that he intended to leave this country's adversaries in uncertainty about its reactions to a challenge. That posture seems by far the stronger one.

The Libyan attack took place over waters that everyone but Libya agrees to be clearly international. Jurisdiction over coastal zones is another matter, even along coasts less fiercely disputed than North Korea's, and when a plane may be flying at two or three times the speed of sound its precise location is not always easy to calculate with absolute precision. Few people in this country or abroad will waste much sympathy on North Korea, which can always be counted on to do its best, within its limited means, to do something disagreeable. North Korea is not a country with a large regard for international law. But it would not be helpful to the United States' larger purposes to let the success of the Libyan encounter--the success in military terms and, at home, political as well--turn into a cliche to form its answer to every challenge on any border.